2023-12-10T18:37:17.028 INFO:root:teuthology version: 0.0.1.dev214+gf2de108 2023-12-10T18:37:17.029 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/dis-2023-12-10_16:10:42-rbd-wip-dis-testing-1-distro-default-smithi/7486362 --name dis-2023-12-10_16:10:42-rbd-wip-dis-testing-1-distro-default-smithi --description rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-stupid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-stress-workunit-minimum} --verbose --owner scheduled_dis@teuthology -- /home/teuthworker/archive/dis-2023-12-10_16:10:42-rbd-wip-dis-testing-1-distro-default-smithi/7486362/orig.config.yaml 2023-12-10T18:37:17.082 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-10T18:37:17.163 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/dis-2023-12-10_16:10:42-rbd-wip-dis-testing-1-distro-default-smithi/7486362 branch: wip-dis-testing-1 description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-stupid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-stress-workunit-minimum} email: idryomov@gmail.com first_in_suite: false job_id: '7486362' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi meta: - desc: run two ceph clusters and install rbd-mirror - desc: configure the permissions for client.mirror - desc: 2 ceph clusters with 1 mon and 3 osds each - desc: run four rbd-mirror daemons per cluster - desc: run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon name: dis-2023-12-10_16:10:42-rbd-wip-dis-testing-1-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 30 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-dis-testing-1 ceph: conf: client: debug rbd: 20 debug rbd_mirror: 15 log to stderr: false rbd default features: 125 rbd mirror image policy type: simple client.mirror.0: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.1: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.2: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.3: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME sha1: 3d2db44fa991bc355074aa9b7647f63b77201fef ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 3d2db44fa991bc355074aa9b7647f63b77201fef thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-dis-testing-1 sha1: 3d2db44fa991bc355074aa9b7647f63b77201fef owner: scheduled_dis@teuthology priority: 74 repo: https://github.com/ceph/ceph-ci roles: - - cluster1.mon.a - cluster1.mgr.x - cluster2.mgr.x - cluster1.osd.0 - cluster1.osd.1 - cluster1.osd.2 - cluster1.client.0 - cluster2.client.0 - - cluster2.mon.a - cluster2.osd.0 - cluster2.osd.1 - cluster2.osd.2 - cluster1.client.mirror - cluster1.client.mirror.0 - cluster1.client.mirror.1 - cluster1.client.mirror.2 - cluster1.client.mirror.3 - cluster1.client.mirror.4 - cluster1.client.mirror.5 - cluster1.client.mirror.6 - cluster2.client.mirror - cluster2.client.mirror.0 - cluster2.client.mirror.1 - cluster2.client.mirror.2 - cluster2.client.mirror.3 - cluster2.client.mirror.4 - cluster2.client.mirror.5 - cluster2.client.mirror.6 seed: 7033 sha1: 3d2db44fa991bc355074aa9b7647f63b77201fef sleep_before_teardown: 0 subset: 91/99 suite: rbd suite_branch: wip-dis-testing-1 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci suite_sha1: 3d2db44fa991bc355074aa9b7647f63b77201fef targets: smithi033.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCg4fpIf9BgAE0K5vY3eaCYNnAdpFkPYg83UNzpXJx+VdnMglbkcDNn9oNHV0h8GlQ68zOzCcQCAjDqUW/H3wNI= smithi076.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDz6mTNSmU7E4IwEUFsmyAfIf0TTvzaqidmq2OKROs5pSoYL6ng2SXq1yuh1NB7AMqvNCBdduJ5kk6F5IVYNexs= tasks: - install: extra_packages: - rbd-mirror - ceph: cluster: cluster1 - ceph: cluster: cluster2 - exec: cluster1.client.mirror.0: - sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - rbd-mirror: client: cluster1.client.mirror.0 thrash: true - rbd-mirror: client: cluster1.client.mirror.1 thrash: true - rbd-mirror: client: cluster1.client.mirror.2 thrash: true - rbd-mirror: client: cluster1.client.mirror.3 thrash: true - rbd-mirror: client: cluster2.client.mirror.0 thrash: true - rbd-mirror: client: cluster2.client.mirror.1 thrash: true - rbd-mirror: client: cluster2.client.mirror.2 thrash: true - rbd-mirror: client: cluster2.client.mirror.3 thrash: true - rbd-mirror-thrash: cluster: cluster1 - rbd-mirror-thrash: cluster: cluster2 - workunit: clients: cluster1.client.mirror: - rbd/rbd_mirror_stress.sh env: CEPH_ARGS: '' MIRROR_POOL_MODE: image RBD_IMAGE_FEATURES: layering RBD_MIRROR_INSTANCES: '4' RBD_MIRROR_MODE: snapshot RBD_MIRROR_USE_EXISTING_CLUSTER: '1' RBD_MIRROR_USE_RBD_MIRROR: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: f2de108beb0a6dde7f3e6c96039071828a1f6269 timestamp: 2023-12-10_16:10:42 tube: smithi user: dis verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.232157 2023-12-10T18:37:17.163 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa; will attempt to use it 2023-12-10T18:37:17.164 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks 2023-12-10T18:37:17.164 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-12-10T18:37:17.166 INFO:teuthology.task.internal:Checking packages... 2023-12-10T18:37:17.188 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '3d2db44fa991bc355074aa9b7647f63b77201fef' 2023-12-10T18:37:17.188 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-10T18:37:17.188 INFO:teuthology.packaging:ref: None 2023-12-10T18:37:17.188 INFO:teuthology.packaging:tag: None 2023-12-10T18:37:17.188 INFO:teuthology.packaging:branch: wip-dis-testing-1 2023-12-10T18:37:17.189 INFO:teuthology.packaging:sha1: 3d2db44fa991bc355074aa9b7647f63b77201fef 2023-12-10T18:37:17.189 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-dis-testing-1 2023-12-10T18:37:17.323 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-130-g3d2db44f-1jammy 2023-12-10T18:37:17.323 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-12-10T18:37:17.330 INFO:teuthology.task.internal:no buildpackages task found 2023-12-10T18:37:17.330 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-12-10T18:37:17.353 INFO:teuthology.task.internal:Saving configuration 2023-12-10T18:37:17.371 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-12-10T18:37:17.378 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-10T18:37:17.403 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi033.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2023-12-10_16:10:42-rbd-wip-dis-testing-1-distro-default-smithi/7486362', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-10 18:30:59.830640', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIO7snHYl/ljzwYUAL8JFffR9qurg2wqpjTnFxmes4tuE'} 2023-12-10T18:37:17.427 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi076.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2023-12-10_16:10:42-rbd-wip-dis-testing-1-distro-default-smithi/7486362', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-10 18:30:59.831797', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIP/QZ3DMSgd6EyA4zlIhnFMqrTTN+qUOQ0rPD5yxUl11'} 2023-12-10T18:37:17.428 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-12-10T18:37:17.434 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'] 2023-12-10T18:37:17.434 INFO:teuthology.task.internal:roles: ubuntu@smithi076.front.sepia.ceph.com - ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6'] 2023-12-10T18:37:17.434 INFO:teuthology.run_tasks:Running task console_log... 2023-12-10T18:37:17.502 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f33a71d8f70>, signals=[15]) 2023-12-10T18:37:17.502 INFO:teuthology.run_tasks:Running task internal.connect... 2023-12-10T18:37:17.508 INFO:teuthology.task.internal:Opening connections... 2023-12-10T18:37:17.509 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi033.front.sepia.ceph.com 2023-12-10T18:37:17.510 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-10T18:37:17.589 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi076.front.sepia.ceph.com 2023-12-10T18:37:17.590 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-10T18:37:17.662 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-12-10T18:37:17.669 DEBUG:teuthology.orchestra.run.smithi033:> uname -m 2023-12-10T18:37:17.674 INFO:teuthology.orchestra.run.smithi033.stdout:x86_64 2023-12-10T18:37:17.674 DEBUG:teuthology.orchestra.run.smithi033:> cat /etc/os-release 2023-12-10T18:37:17.722 INFO:teuthology.orchestra.run.smithi033.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2023-12-10T18:37:17.722 INFO:teuthology.orchestra.run.smithi033.stdout:NAME="Ubuntu" 2023-12-10T18:37:17.722 INFO:teuthology.orchestra.run.smithi033.stdout:VERSION_ID="22.04" 2023-12-10T18:37:17.722 INFO:teuthology.orchestra.run.smithi033.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2023-12-10T18:37:17.722 INFO:teuthology.orchestra.run.smithi033.stdout:VERSION_CODENAME=jammy 2023-12-10T18:37:17.722 INFO:teuthology.orchestra.run.smithi033.stdout:ID=ubuntu 2023-12-10T18:37:17.722 INFO:teuthology.orchestra.run.smithi033.stdout:ID_LIKE=debian 2023-12-10T18:37:17.723 INFO:teuthology.orchestra.run.smithi033.stdout:HOME_URL="https://www.ubuntu.com/" 2023-12-10T18:37:17.723 INFO:teuthology.orchestra.run.smithi033.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-12-10T18:37:17.723 INFO:teuthology.orchestra.run.smithi033.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-12-10T18:37:17.723 INFO:teuthology.orchestra.run.smithi033.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-12-10T18:37:17.723 INFO:teuthology.orchestra.run.smithi033.stdout:UBUNTU_CODENAME=jammy 2023-12-10T18:37:17.723 INFO:teuthology.lock.ops:Updating smithi033.front.sepia.ceph.com on lock server 2023-12-10T18:37:17.751 DEBUG:teuthology.orchestra.run.smithi076:> uname -m 2023-12-10T18:37:17.756 INFO:teuthology.orchestra.run.smithi076.stdout:x86_64 2023-12-10T18:37:17.756 DEBUG:teuthology.orchestra.run.smithi076:> cat /etc/os-release 2023-12-10T18:37:17.804 INFO:teuthology.orchestra.run.smithi076.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2023-12-10T18:37:17.804 INFO:teuthology.orchestra.run.smithi076.stdout:NAME="Ubuntu" 2023-12-10T18:37:17.804 INFO:teuthology.orchestra.run.smithi076.stdout:VERSION_ID="22.04" 2023-12-10T18:37:17.804 INFO:teuthology.orchestra.run.smithi076.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2023-12-10T18:37:17.804 INFO:teuthology.orchestra.run.smithi076.stdout:VERSION_CODENAME=jammy 2023-12-10T18:37:17.805 INFO:teuthology.orchestra.run.smithi076.stdout:ID=ubuntu 2023-12-10T18:37:17.805 INFO:teuthology.orchestra.run.smithi076.stdout:ID_LIKE=debian 2023-12-10T18:37:17.805 INFO:teuthology.orchestra.run.smithi076.stdout:HOME_URL="https://www.ubuntu.com/" 2023-12-10T18:37:17.805 INFO:teuthology.orchestra.run.smithi076.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-12-10T18:37:17.805 INFO:teuthology.orchestra.run.smithi076.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-12-10T18:37:17.805 INFO:teuthology.orchestra.run.smithi076.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-12-10T18:37:17.805 INFO:teuthology.orchestra.run.smithi076.stdout:UBUNTU_CODENAME=jammy 2023-12-10T18:37:17.805 INFO:teuthology.lock.ops:Updating smithi076.front.sepia.ceph.com on lock server 2023-12-10T18:37:17.828 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-12-10T18:37:17.838 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-12-10T18:37:17.844 INFO:teuthology.task.internal:Checking for old test directory... 2023-12-10T18:37:17.844 DEBUG:teuthology.orchestra.run.smithi033:> test '!' -e /home/ubuntu/cephtest 2023-12-10T18:37:17.847 DEBUG:teuthology.orchestra.run.smithi076:> test '!' -e /home/ubuntu/cephtest 2023-12-10T18:37:17.851 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-12-10T18:37:17.857 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-12-10T18:37:17.857 DEBUG:teuthology.orchestra.run.smithi033:> test -z $(ls -A /var/lib/ceph) 2023-12-10T18:37:17.892 DEBUG:teuthology.orchestra.run.smithi076:> test -z $(ls -A /var/lib/ceph) 2023-12-10T18:37:17.911 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-12-10T18:37:17.957 INFO:teuthology.run_tasks:Running task kernel... 2023-12-10T18:37:17.972 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-12-10T18:37:17.972 INFO:teuthology.task.kernel:config {'cluster1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'cluster2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-12-10T18:37:17.972 DEBUG:teuthology.orchestra.run.smithi033:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-10T18:37:17.973 DEBUG:teuthology.orchestra.run.smithi076:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-10T18:37:17.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-10T18:37:17.977 DEBUG:teuthology.orchestra.run.smithi033:> uname -r 2023-12-10T18:37:17.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-10T18:37:17.979 DEBUG:teuthology.orchestra.run.smithi076:> uname -r 2023-12-10T18:37:18.023 INFO:teuthology.orchestra.run.smithi033.stdout:5.15.0-56-generic 2023-12-10T18:37:18.023 INFO:teuthology.task.kernel:Running kernel on smithi033: 5.15.0-56-generic 2023-12-10T18:37:18.023 DEBUG:teuthology.orchestra.run.smithi033:> sudo apt-get clean 2023-12-10T18:37:18.025 INFO:teuthology.orchestra.run.smithi076.stdout:5.15.0-56-generic 2023-12-10T18:37:18.025 INFO:teuthology.task.kernel:Running kernel on smithi076: 5.15.0-56-generic 2023-12-10T18:37:18.025 DEBUG:teuthology.orchestra.run.smithi076:> sudo apt-get clean 2023-12-10T18:37:18.146 DEBUG:teuthology.orchestra.run.smithi033:> sudo apt-get update 2023-12-10T18:37:18.155 DEBUG:teuthology.orchestra.run.smithi076:> sudo apt-get update 2023-12-10T18:37:18.291 INFO:teuthology.orchestra.run.smithi076.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-12-10T18:37:18.291 INFO:teuthology.orchestra.run.smithi076.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2023-12-10T18:37:18.295 INFO:teuthology.orchestra.run.smithi076.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2023-12-10T18:37:18.383 INFO:teuthology.orchestra.run.smithi076.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2023-12-10T18:37:18.433 INFO:teuthology.orchestra.run.smithi033.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-12-10T18:37:18.433 INFO:teuthology.orchestra.run.smithi033.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2023-12-10T18:37:18.494 INFO:teuthology.orchestra.run.smithi033.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2023-12-10T18:37:18.516 INFO:teuthology.orchestra.run.smithi076.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [376 kB] 2023-12-10T18:37:18.602 INFO:teuthology.orchestra.run.smithi076.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,016 kB] 2023-12-10T18:37:18.604 INFO:teuthology.orchestra.run.smithi076.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,244 kB] 2023-12-10T18:37:18.633 INFO:teuthology.orchestra.run.smithi076.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [195 kB] 2023-12-10T18:37:18.637 INFO:teuthology.orchestra.run.smithi076.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2023-12-10T18:37:18.659 INFO:teuthology.orchestra.run.smithi076.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,179 kB] 2023-12-10T18:37:18.666 INFO:teuthology.orchestra.run.smithi076.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [32.4 kB] 2023-12-10T18:37:18.667 INFO:teuthology.orchestra.run.smithi076.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [191 kB] 2023-12-10T18:37:18.670 INFO:teuthology.orchestra.run.smithi076.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2023-12-10T18:37:18.670 INFO:teuthology.orchestra.run.smithi076.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [577 kB] 2023-12-10T18:37:18.672 INFO:teuthology.orchestra.run.smithi076.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [815 kB] 2023-12-10T18:37:18.678 INFO:teuthology.orchestra.run.smithi076.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [152 kB] 2023-12-10T18:37:18.679 INFO:teuthology.orchestra.run.smithi076.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2023-12-10T18:37:18.680 INFO:teuthology.orchestra.run.smithi076.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [36.5 kB] 2023-12-10T18:37:18.693 INFO:teuthology.orchestra.run.smithi076.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,032 B] 2023-12-10T18:37:18.693 INFO:teuthology.orchestra.run.smithi076.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,060 B] 2023-12-10T18:37:18.693 INFO:teuthology.orchestra.run.smithi076.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2023-12-10T18:37:18.701 INFO:teuthology.orchestra.run.smithi076.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [543 kB] 2023-12-10T18:37:18.712 INFO:teuthology.orchestra.run.smithi076.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [257 kB] 2023-12-10T18:37:18.717 INFO:teuthology.orchestra.run.smithi076.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2023-12-10T18:37:18.717 INFO:teuthology.orchestra.run.smithi076.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [32.8 kB] 2023-12-10T18:37:18.719 INFO:teuthology.orchestra.run.smithi076.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,226 kB] 2023-12-10T18:37:18.744 INFO:teuthology.orchestra.run.smithi076.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [199 kB] 2023-12-10T18:37:18.748 INFO:teuthology.orchestra.run.smithi076.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2023-12-10T18:37:18.749 INFO:teuthology.orchestra.run.smithi076.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,018 kB] 2023-12-10T18:37:18.773 INFO:teuthology.orchestra.run.smithi076.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [675 kB] 2023-12-10T18:37:18.782 INFO:teuthology.orchestra.run.smithi076.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [226 kB] 2023-12-10T18:37:18.786 INFO:teuthology.orchestra.run.smithi076.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2023-12-10T18:37:18.792 INFO:teuthology.orchestra.run.smithi076.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [41.6 kB] 2023-12-10T18:37:18.793 INFO:teuthology.orchestra.run.smithi076.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [3,888 B] 2023-12-10T18:37:18.793 INFO:teuthology.orchestra.run.smithi076.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [9,768 B] 2023-12-10T18:37:18.793 INFO:teuthology.orchestra.run.smithi076.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2023-12-10T18:37:18.794 INFO:teuthology.orchestra.run.smithi076.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [56.3 kB] 2023-12-10T18:37:18.794 INFO:teuthology.orchestra.run.smithi076.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [64.2 kB] 2023-12-10T18:37:18.824 INFO:teuthology.orchestra.run.smithi076.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [10.5 kB] 2023-12-10T18:37:18.824 INFO:teuthology.orchestra.run.smithi076.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2023-12-10T18:37:18.825 INFO:teuthology.orchestra.run.smithi076.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.8 kB] 2023-12-10T18:37:18.825 INFO:teuthology.orchestra.run.smithi076.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.8 kB] 2023-12-10T18:37:18.847 INFO:teuthology.orchestra.run.smithi033.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2023-12-10T18:37:18.851 INFO:teuthology.orchestra.run.smithi033.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [376 kB] 2023-12-10T18:37:18.854 INFO:teuthology.orchestra.run.smithi076.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2023-12-10T18:37:18.855 INFO:teuthology.orchestra.run.smithi076.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2023-12-10T18:37:18.986 INFO:teuthology.orchestra.run.smithi033.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,244 kB] 2023-12-10T18:37:19.127 INFO:teuthology.orchestra.run.smithi033.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,016 kB] 2023-12-10T18:37:19.270 INFO:teuthology.orchestra.run.smithi033.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [543 kB] 2023-12-10T18:37:19.279 INFO:teuthology.orchestra.run.smithi033.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [195 kB] 2023-12-10T18:37:19.297 INFO:teuthology.orchestra.run.smithi033.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2023-12-10T18:37:19.298 INFO:teuthology.orchestra.run.smithi033.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [32.4 kB] 2023-12-10T18:37:19.300 INFO:teuthology.orchestra.run.smithi033.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,179 kB] 2023-12-10T18:37:19.308 INFO:teuthology.orchestra.run.smithi033.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [257 kB] 2023-12-10T18:37:19.325 INFO:teuthology.orchestra.run.smithi033.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2023-12-10T18:37:19.327 INFO:teuthology.orchestra.run.smithi033.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [32.8 kB] 2023-12-10T18:37:19.328 INFO:teuthology.orchestra.run.smithi033.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,226 kB] 2023-12-10T18:37:19.371 INFO:teuthology.orchestra.run.smithi033.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [191 kB] 2023-12-10T18:37:19.413 INFO:teuthology.orchestra.run.smithi033.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2023-12-10T18:37:19.413 INFO:teuthology.orchestra.run.smithi033.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [199 kB] 2023-12-10T18:37:19.414 INFO:teuthology.orchestra.run.smithi033.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [577 kB] 2023-12-10T18:37:19.419 INFO:teuthology.orchestra.run.smithi033.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2023-12-10T18:37:19.427 INFO:teuthology.orchestra.run.smithi033.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [815 kB] 2023-12-10T18:37:19.585 INFO:teuthology.orchestra.run.smithi033.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [675 kB] 2023-12-10T18:37:19.605 INFO:teuthology.orchestra.run.smithi033.stdout:Get:24 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [152 kB] 2023-12-10T18:37:19.993 INFO:teuthology.orchestra.run.smithi033.stdout:Get:25 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2023-12-10T18:37:20.000 INFO:teuthology.orchestra.run.smithi033.stdout:Get:26 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,032 B] 2023-12-10T18:37:20.000 INFO:teuthology.orchestra.run.smithi033.stdout:Get:27 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [36.5 kB] 2023-12-10T18:37:20.014 INFO:teuthology.orchestra.run.smithi033.stdout:Get:28 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,060 B] 2023-12-10T18:37:20.018 INFO:teuthology.orchestra.run.smithi033.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2023-12-10T18:37:20.161 INFO:teuthology.orchestra.run.smithi033.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,018 kB] 2023-12-10T18:37:20.270 INFO:teuthology.orchestra.run.smithi033.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [226 kB] 2023-12-10T18:37:20.290 INFO:teuthology.orchestra.run.smithi033.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2023-12-10T18:37:20.291 INFO:teuthology.orchestra.run.smithi033.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [41.6 kB] 2023-12-10T18:37:20.295 INFO:teuthology.orchestra.run.smithi033.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [3,888 B] 2023-12-10T18:37:20.295 INFO:teuthology.orchestra.run.smithi033.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [9,768 B] 2023-12-10T18:37:20.296 INFO:teuthology.orchestra.run.smithi033.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2023-12-10T18:37:20.297 INFO:teuthology.orchestra.run.smithi033.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [64.2 kB] 2023-12-10T18:37:20.301 INFO:teuthology.orchestra.run.smithi033.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [56.3 kB] 2023-12-10T18:37:20.304 INFO:teuthology.orchestra.run.smithi033.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [10.5 kB] 2023-12-10T18:37:20.469 INFO:teuthology.orchestra.run.smithi033.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2023-12-10T18:37:20.551 INFO:teuthology.orchestra.run.smithi033.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.8 kB] 2023-12-10T18:37:20.658 INFO:teuthology.orchestra.run.smithi033.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.8 kB] 2023-12-10T18:37:20.716 INFO:teuthology.orchestra.run.smithi033.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2023-12-10T18:37:20.750 INFO:teuthology.orchestra.run.smithi033.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2023-12-10T18:37:24.101 INFO:teuthology.orchestra.run.smithi076.stdout:Fetched 10.7 MB in 2s (6,996 kB/s) 2023-12-10T18:37:25.240 INFO:teuthology.orchestra.run.smithi033.stdout:Fetched 10.7 MB in 3s (4,223 kB/s) 2023-12-10T18:37:25.487 INFO:teuthology.orchestra.run.smithi076.stdout:Reading package lists... 2023-12-10T18:37:25.506 DEBUG:teuthology.orchestra.run.smithi076:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-12-10T18:37:25.566 INFO:teuthology.orchestra.run.smithi076.stdout:Reading package lists... 2023-12-10T18:37:25.736 INFO:teuthology.orchestra.run.smithi076.stdout:Building dependency tree... 2023-12-10T18:37:25.737 INFO:teuthology.orchestra.run.smithi076.stdout:Reading state information... 2023-12-10T18:37:25.880 INFO:teuthology.orchestra.run.smithi076.stdout:The following packages were automatically installed and are no longer required: 2023-12-10T18:37:25.881 INFO:teuthology.orchestra.run.smithi076.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-12-10T18:37:25.881 INFO:teuthology.orchestra.run.smithi076.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-12-10T18:37:25.881 INFO:teuthology.orchestra.run.smithi076.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-12-10T18:37:25.881 INFO:teuthology.orchestra.run.smithi076.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2023-12-10T18:37:25.881 INFO:teuthology.orchestra.run.smithi076.stdout: python2.7-minimal 2023-12-10T18:37:25.882 INFO:teuthology.orchestra.run.smithi076.stdout:Use 'sudo apt autoremove' to remove them. 2023-12-10T18:37:25.882 INFO:teuthology.orchestra.run.smithi076.stdout:The following additional packages will be installed: 2023-12-10T18:37:25.882 INFO:teuthology.orchestra.run.smithi076.stdout: linux-generic linux-headers-5.15.0-91 linux-headers-5.15.0-91-generic 2023-12-10T18:37:25.882 INFO:teuthology.orchestra.run.smithi076.stdout: linux-headers-generic linux-image-5.15.0-91-generic 2023-12-10T18:37:25.883 INFO:teuthology.orchestra.run.smithi076.stdout: linux-modules-5.15.0-91-generic linux-modules-extra-5.15.0-91-generic 2023-12-10T18:37:25.884 INFO:teuthology.orchestra.run.smithi076.stdout:Suggested packages: 2023-12-10T18:37:25.884 INFO:teuthology.orchestra.run.smithi076.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2023-12-10T18:37:25.884 INFO:teuthology.orchestra.run.smithi076.stdout:Recommended packages: 2023-12-10T18:37:25.884 INFO:teuthology.orchestra.run.smithi076.stdout: thermald 2023-12-10T18:37:25.919 INFO:teuthology.orchestra.run.smithi076.stdout:The following NEW packages will be installed: 2023-12-10T18:37:25.920 INFO:teuthology.orchestra.run.smithi076.stdout: linux-headers-5.15.0-91 linux-headers-5.15.0-91-generic 2023-12-10T18:37:25.920 INFO:teuthology.orchestra.run.smithi076.stdout: linux-image-5.15.0-91-generic linux-modules-5.15.0-91-generic 2023-12-10T18:37:25.920 INFO:teuthology.orchestra.run.smithi076.stdout: linux-modules-extra-5.15.0-91-generic 2023-12-10T18:37:25.920 INFO:teuthology.orchestra.run.smithi076.stdout:The following packages will be upgraded: 2023-12-10T18:37:25.921 INFO:teuthology.orchestra.run.smithi076.stdout: linux-generic linux-headers-generic linux-image-generic 2023-12-10T18:37:25.973 INFO:teuthology.orchestra.run.smithi076.stdout:3 upgraded, 5 newly installed, 0 to remove and 275 not upgraded. 2023-12-10T18:37:25.973 INFO:teuthology.orchestra.run.smithi076.stdout:Need to get 113 MB of archives. 2023-12-10T18:37:25.973 INFO:teuthology.orchestra.run.smithi076.stdout:After this operation, 582 MB of additional disk space will be used. 2023-12-10T18:37:25.973 INFO:teuthology.orchestra.run.smithi076.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-91-generic amd64 5.15.0-91.101 [22.5 MB] 2023-12-10T18:37:26.601 INFO:teuthology.orchestra.run.smithi033.stdout:Reading package lists... 2023-12-10T18:37:26.619 DEBUG:teuthology.orchestra.run.smithi033:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-12-10T18:37:26.676 INFO:teuthology.orchestra.run.smithi033.stdout:Reading package lists... 2023-12-10T18:37:26.779 INFO:teuthology.orchestra.run.smithi076.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-91-generic amd64 5.15.0-91.101 [11.5 MB] 2023-12-10T18:37:26.869 INFO:teuthology.orchestra.run.smithi033.stdout:Building dependency tree... 2023-12-10T18:37:26.869 INFO:teuthology.orchestra.run.smithi033.stdout:Reading state information... 2023-12-10T18:37:26.953 INFO:teuthology.orchestra.run.smithi076.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-91-generic amd64 5.15.0-91.101 [63.8 MB] 2023-12-10T18:37:27.041 INFO:teuthology.orchestra.run.smithi033.stdout:The following packages were automatically installed and are no longer required: 2023-12-10T18:37:27.041 INFO:teuthology.orchestra.run.smithi033.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-12-10T18:37:27.042 INFO:teuthology.orchestra.run.smithi033.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-12-10T18:37:27.042 INFO:teuthology.orchestra.run.smithi033.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-12-10T18:37:27.042 INFO:teuthology.orchestra.run.smithi033.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2023-12-10T18:37:27.042 INFO:teuthology.orchestra.run.smithi033.stdout: python2.7-minimal 2023-12-10T18:37:27.042 INFO:teuthology.orchestra.run.smithi033.stdout:Use 'sudo apt autoremove' to remove them. 2023-12-10T18:37:27.043 INFO:teuthology.orchestra.run.smithi033.stdout:The following additional packages will be installed: 2023-12-10T18:37:27.043 INFO:teuthology.orchestra.run.smithi033.stdout: linux-generic linux-headers-5.15.0-91 linux-headers-5.15.0-91-generic 2023-12-10T18:37:27.043 INFO:teuthology.orchestra.run.smithi033.stdout: linux-headers-generic linux-image-5.15.0-91-generic 2023-12-10T18:37:27.044 INFO:teuthology.orchestra.run.smithi033.stdout: linux-modules-5.15.0-91-generic linux-modules-extra-5.15.0-91-generic 2023-12-10T18:37:27.045 INFO:teuthology.orchestra.run.smithi033.stdout:Suggested packages: 2023-12-10T18:37:27.045 INFO:teuthology.orchestra.run.smithi033.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2023-12-10T18:37:27.045 INFO:teuthology.orchestra.run.smithi033.stdout:Recommended packages: 2023-12-10T18:37:27.045 INFO:teuthology.orchestra.run.smithi033.stdout: thermald 2023-12-10T18:37:27.091 INFO:teuthology.orchestra.run.smithi033.stdout:The following NEW packages will be installed: 2023-12-10T18:37:27.092 INFO:teuthology.orchestra.run.smithi033.stdout: linux-headers-5.15.0-91 linux-headers-5.15.0-91-generic 2023-12-10T18:37:27.092 INFO:teuthology.orchestra.run.smithi033.stdout: linux-image-5.15.0-91-generic linux-modules-5.15.0-91-generic 2023-12-10T18:37:27.092 INFO:teuthology.orchestra.run.smithi033.stdout: linux-modules-extra-5.15.0-91-generic 2023-12-10T18:37:27.093 INFO:teuthology.orchestra.run.smithi033.stdout:The following packages will be upgraded: 2023-12-10T18:37:27.094 INFO:teuthology.orchestra.run.smithi033.stdout: linux-generic linux-headers-generic linux-image-generic 2023-12-10T18:37:27.274 INFO:teuthology.orchestra.run.smithi033.stdout:3 upgraded, 5 newly installed, 0 to remove and 275 not upgraded. 2023-12-10T18:37:27.274 INFO:teuthology.orchestra.run.smithi033.stdout:Need to get 113 MB of archives. 2023-12-10T18:37:27.274 INFO:teuthology.orchestra.run.smithi033.stdout:After this operation, 582 MB of additional disk space will be used. 2023-12-10T18:37:27.274 INFO:teuthology.orchestra.run.smithi033.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-91-generic amd64 5.15.0-91.101 [22.5 MB] 2023-12-10T18:37:28.191 INFO:teuthology.orchestra.run.smithi076.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.91.88 [1,700 B] 2023-12-10T18:37:28.191 INFO:teuthology.orchestra.run.smithi076.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.91.88 [2,488 B] 2023-12-10T18:37:28.192 INFO:teuthology.orchestra.run.smithi076.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-91 all 5.15.0-91.101 [12.3 MB] 2023-12-10T18:37:28.685 INFO:teuthology.orchestra.run.smithi033.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-91-generic amd64 5.15.0-91.101 [11.5 MB] 2023-12-10T18:37:28.850 INFO:teuthology.orchestra.run.smithi076.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-91-generic amd64 5.15.0-91.101 [2,860 kB] 2023-12-10T18:37:28.979 INFO:teuthology.orchestra.run.smithi076.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.91.88 [2,342 B] 2023-12-10T18:37:29.021 INFO:teuthology.orchestra.run.smithi033.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-91-generic amd64 5.15.0-91.101 [63.8 MB] 2023-12-10T18:37:29.390 INFO:teuthology.orchestra.run.smithi076.stdout:Fetched 113 MB in 3s (37.0 MB/s) 2023-12-10T18:37:29.588 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package linux-modules-5.15.0-91-generic. 2023-12-10T18:37:31.204 INFO:teuthology.orchestra.run.smithi033.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.91.88 [1,700 B] 2023-12-10T18:37:31.250 INFO:teuthology.orchestra.run.smithi033.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.91.88 [2,488 B] 2023-12-10T18:37:31.251 INFO:teuthology.orchestra.run.smithi033.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-91 all 5.15.0-91.101 [12.3 MB] 2023-12-10T18:37:31.596 INFO:teuthology.orchestra.run.smithi076.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2023-12-10T18:37:31.600 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../0-linux-modules-5.15.0-91-generic_5.15.0-91.101_amd64.deb ... 2023-12-10T18:37:31.727 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking linux-modules-5.15.0-91-generic (5.15.0-91.101) ... 2023-12-10T18:37:31.864 INFO:teuthology.orchestra.run.smithi033.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-91-generic amd64 5.15.0-91.101 [2,860 kB] 2023-12-10T18:37:32.024 INFO:teuthology.orchestra.run.smithi033.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.91.88 [2,342 B] 2023-12-10T18:37:32.473 INFO:teuthology.orchestra.run.smithi033.stdout:Fetched 113 MB in 5s (22.9 MB/s) 2023-12-10T18:37:32.660 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package linux-modules-5.15.0-91-generic. 2023-12-10T18:37:34.045 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package linux-image-5.15.0-91-generic. 2023-12-10T18:37:34.066 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../1-linux-image-5.15.0-91-generic_5.15.0-91.101_amd64.deb ... 2023-12-10T18:37:34.153 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking linux-image-5.15.0-91-generic (5.15.0-91.101) ... 2023-12-10T18:37:34.677 INFO:teuthology.orchestra.run.smithi033.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2023-12-10T18:37:34.682 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../0-linux-modules-5.15.0-91-generic_5.15.0-91.101_amd64.deb ... 2023-12-10T18:37:34.775 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking linux-modules-5.15.0-91-generic (5.15.0-91.101) ... 2023-12-10T18:37:34.974 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-91-generic. 2023-12-10T18:37:34.994 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-91-generic_5.15.0-91.101_amd64.deb ... 2023-12-10T18:37:35.032 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking linux-modules-extra-5.15.0-91-generic (5.15.0-91.101) ... 2023-12-10T18:37:37.585 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package linux-image-5.15.0-91-generic. 2023-12-10T18:37:37.607 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../1-linux-image-5.15.0-91-generic_5.15.0-91.101_amd64.deb ... 2023-12-10T18:37:37.708 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking linux-image-5.15.0-91-generic (5.15.0-91.101) ... 2023-12-10T18:37:38.214 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-91-generic. 2023-12-10T18:37:38.234 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-91-generic_5.15.0-91.101_amd64.deb ... 2023-12-10T18:37:38.272 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking linux-modules-extra-5.15.0-91-generic (5.15.0-91.101) ... 2023-12-10T18:37:41.616 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../3-linux-generic_5.15.0.91.88_amd64.deb ... 2023-12-10T18:37:41.746 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking linux-generic (5.15.0.91.88) over (5.15.0.56.54) ... 2023-12-10T18:37:42.076 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.91.88_amd64.deb ... 2023-12-10T18:37:42.180 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking linux-image-generic (5.15.0.91.88) over (5.15.0.56.54) ... 2023-12-10T18:37:42.458 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package linux-headers-5.15.0-91. 2023-12-10T18:37:42.469 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../5-linux-headers-5.15.0-91_5.15.0-91.101_all.deb ... 2023-12-10T18:37:42.508 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking linux-headers-5.15.0-91 (5.15.0-91.101) ... 2023-12-10T18:37:44.842 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../3-linux-generic_5.15.0.91.88_amd64.deb ... 2023-12-10T18:37:45.001 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking linux-generic (5.15.0.91.88) over (5.15.0.56.54) ... 2023-12-10T18:37:45.326 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.91.88_amd64.deb ... 2023-12-10T18:37:45.455 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking linux-image-generic (5.15.0.91.88) over (5.15.0.56.54) ... 2023-12-10T18:37:45.760 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package linux-headers-5.15.0-91. 2023-12-10T18:37:45.783 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../5-linux-headers-5.15.0-91_5.15.0-91.101_all.deb ... 2023-12-10T18:37:45.827 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking linux-headers-5.15.0-91 (5.15.0-91.101) ... 2023-12-10T18:37:48.107 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package linux-headers-5.15.0-91-generic. 2023-12-10T18:37:48.143 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../6-linux-headers-5.15.0-91-generic_5.15.0-91.101_amd64.deb ... 2023-12-10T18:37:48.182 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking linux-headers-5.15.0-91-generic (5.15.0-91.101) ... 2023-12-10T18:37:51.467 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.91.88_amd64.deb ... 2023-12-10T18:37:51.590 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking linux-headers-generic (5.15.0.91.88) over (5.15.0.56.54) ... 2023-12-10T18:37:51.756 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package linux-headers-5.15.0-91-generic. 2023-12-10T18:37:51.793 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../6-linux-headers-5.15.0-91-generic_5.15.0-91.101_amd64.deb ... 2023-12-10T18:37:51.839 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking linux-headers-5.15.0-91-generic (5.15.0-91.101) ... 2023-12-10T18:37:51.932 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up linux-headers-5.15.0-91 (5.15.0-91.101) ... 2023-12-10T18:37:52.053 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up linux-headers-5.15.0-91-generic (5.15.0-91.101) ... 2023-12-10T18:37:52.162 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up linux-headers-generic (5.15.0.91.88) ... 2023-12-10T18:37:52.291 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up linux-image-5.15.0-91-generic (5.15.0-91.101) ... 2023-12-10T18:37:53.610 INFO:teuthology.orchestra.run.smithi076.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2023-12-10T18:37:53.610 INFO:teuthology.orchestra.run.smithi076.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2023-12-10T18:37:53.610 INFO:teuthology.orchestra.run.smithi076.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-91-generic 2023-12-10T18:37:53.610 INFO:teuthology.orchestra.run.smithi076.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-91-generic 2023-12-10T18:37:53.775 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up linux-modules-extra-5.15.0-91-generic (5.15.0-91.101) ... 2023-12-10T18:37:55.269 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.91.88_amd64.deb ... 2023-12-10T18:37:55.416 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking linux-headers-generic (5.15.0.91.88) over (5.15.0.56.54) ... 2023-12-10T18:37:55.460 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up linux-image-generic (5.15.0.91.88) ... 2023-12-10T18:37:55.561 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up linux-modules-5.15.0-91-generic (5.15.0-91.101) ... 2023-12-10T18:37:55.766 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up linux-headers-5.15.0-91 (5.15.0-91.101) ... 2023-12-10T18:37:55.914 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up linux-headers-5.15.0-91-generic (5.15.0-91.101) ... 2023-12-10T18:37:56.038 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up linux-headers-generic (5.15.0.91.88) ... 2023-12-10T18:37:56.167 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up linux-image-5.15.0-91-generic (5.15.0-91.101) ... 2023-12-10T18:37:56.929 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up linux-generic (5.15.0.91.88) ... 2023-12-10T18:37:57.055 INFO:teuthology.orchestra.run.smithi076.stdout:Processing triggers for linux-image-5.15.0-91-generic (5.15.0-91.101) ... 2023-12-10T18:37:57.119 INFO:teuthology.orchestra.run.smithi076.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-12-10T18:37:57.119 INFO:teuthology.orchestra.run.smithi076.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-91-generic 2023-12-10T18:37:57.517 INFO:teuthology.orchestra.run.smithi033.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2023-12-10T18:37:57.517 INFO:teuthology.orchestra.run.smithi033.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2023-12-10T18:37:57.517 INFO:teuthology.orchestra.run.smithi033.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-91-generic 2023-12-10T18:37:57.517 INFO:teuthology.orchestra.run.smithi033.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-91-generic 2023-12-10T18:37:57.694 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up linux-modules-extra-5.15.0-91-generic (5.15.0-91.101) ... 2023-12-10T18:37:59.137 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up linux-image-generic (5.15.0.91.88) ... 2023-12-10T18:37:59.264 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up linux-modules-5.15.0-91-generic (5.15.0-91.101) ... 2023-12-10T18:38:00.731 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up linux-generic (5.15.0.91.88) ... 2023-12-10T18:38:00.857 INFO:teuthology.orchestra.run.smithi033.stdout:Processing triggers for linux-image-5.15.0-91-generic (5.15.0-91.101) ... 2023-12-10T18:38:00.924 INFO:teuthology.orchestra.run.smithi033.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-12-10T18:38:00.924 INFO:teuthology.orchestra.run.smithi033.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-91-generic 2023-12-10T18:38:11.149 INFO:teuthology.orchestra.run.smithi076.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-12-10T18:38:11.149 INFO:teuthology.orchestra.run.smithi076.stdout:Sourcing file `/etc/default/grub' 2023-12-10T18:38:11.159 INFO:teuthology.orchestra.run.smithi076.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-12-10T18:38:11.178 INFO:teuthology.orchestra.run.smithi076.stdout:Generating grub configuration file ... 2023-12-10T18:38:11.462 INFO:teuthology.orchestra.run.smithi076.stdout:Found linux image: /boot/vmlinuz-5.15.0-91-generic 2023-12-10T18:38:11.481 INFO:teuthology.orchestra.run.smithi076.stdout:Found initrd image: /boot/initrd.img-5.15.0-91-generic 2023-12-10T18:38:11.790 INFO:teuthology.orchestra.run.smithi076.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-12-10T18:38:11.796 INFO:teuthology.orchestra.run.smithi076.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-12-10T18:38:11.862 INFO:teuthology.orchestra.run.smithi076.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-12-10T18:38:11.867 INFO:teuthology.orchestra.run.smithi076.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-12-10T18:38:12.048 INFO:teuthology.orchestra.run.smithi076.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2023-12-10T18:38:12.048 INFO:teuthology.orchestra.run.smithi076.stdout:Systems on them will not be added to the GRUB boot configuration. 2023-12-10T18:38:12.048 INFO:teuthology.orchestra.run.smithi076.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-12-10T18:38:12.088 INFO:teuthology.orchestra.run.smithi076.stdout:done 2023-12-10T18:38:12.543 DEBUG:teuthology.orchestra.run.smithi076:> dpkg -s linux-image-generic 2023-12-10T18:38:12.561 INFO:teuthology.orchestra.run.smithi076.stdout:Package: linux-image-generic 2023-12-10T18:38:12.561 INFO:teuthology.orchestra.run.smithi076.stdout:Status: install ok installed 2023-12-10T18:38:12.561 INFO:teuthology.orchestra.run.smithi076.stdout:Priority: optional 2023-12-10T18:38:12.561 INFO:teuthology.orchestra.run.smithi076.stdout:Section: kernel 2023-12-10T18:38:12.562 INFO:teuthology.orchestra.run.smithi076.stdout:Installed-Size: 20 2023-12-10T18:38:12.562 INFO:teuthology.orchestra.run.smithi076.stdout:Maintainer: Ubuntu Kernel Team 2023-12-10T18:38:12.562 INFO:teuthology.orchestra.run.smithi076.stdout:Architecture: amd64 2023-12-10T18:38:12.562 INFO:teuthology.orchestra.run.smithi076.stdout:Source: linux-meta 2023-12-10T18:38:12.562 INFO:teuthology.orchestra.run.smithi076.stdout:Version: 5.15.0.91.88 2023-12-10T18:38:12.562 INFO:teuthology.orchestra.run.smithi076.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-91), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-12-10T18:38:12.562 INFO:teuthology.orchestra.run.smithi076.stdout:Depends: linux-image-5.15.0-91-generic, linux-modules-extra-5.15.0-91-generic, linux-firmware, intel-microcode, amd64-microcode 2023-12-10T18:38:12.562 INFO:teuthology.orchestra.run.smithi076.stdout:Recommends: thermald 2023-12-10T18:38:12.563 INFO:teuthology.orchestra.run.smithi076.stdout:Description: Generic Linux kernel image 2023-12-10T18:38:12.563 INFO:teuthology.orchestra.run.smithi076.stdout: This package will always depend on the latest generic kernel image 2023-12-10T18:38:12.563 INFO:teuthology.orchestra.run.smithi076.stdout: available. 2023-12-10T18:38:12.563 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-91-generic 2023-12-10T18:38:12.563 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-10T18:38:12.564 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-10T18:38:12.564 INFO:teuthology.task.kernel:Installing distro kernel on cluster2.mon.a... 2023-12-10T18:38:12.564 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi076.front.sepia.ceph.com, path=None, version=distro) 2023-12-10T18:38:12.564 DEBUG:teuthology.orchestra.run.smithi076:> sudo apt-get clean 2023-12-10T18:38:12.695 DEBUG:teuthology.orchestra.run.smithi076:> sudo apt-get update 2023-12-10T18:38:12.801 INFO:teuthology.orchestra.run.smithi076.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-12-10T18:38:12.811 INFO:teuthology.orchestra.run.smithi076.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-12-10T18:38:12.816 INFO:teuthology.orchestra.run.smithi076.stdout:Hit:3 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-12-10T18:38:12.828 INFO:teuthology.orchestra.run.smithi076.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-12-10T18:38:14.117 INFO:teuthology.orchestra.run.smithi076.stdout:Reading package lists... 2023-12-10T18:38:14.135 DEBUG:teuthology.orchestra.run.smithi076:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-12-10T18:38:14.191 INFO:teuthology.orchestra.run.smithi076.stdout:Reading package lists... 2023-12-10T18:38:14.377 INFO:teuthology.orchestra.run.smithi076.stdout:Building dependency tree... 2023-12-10T18:38:14.378 INFO:teuthology.orchestra.run.smithi076.stdout:Reading state information... 2023-12-10T18:38:14.538 INFO:teuthology.orchestra.run.smithi076.stdout:linux-image-generic is already the newest version (5.15.0.91.88). 2023-12-10T18:38:14.539 INFO:teuthology.orchestra.run.smithi076.stdout:The following packages were automatically installed and are no longer required: 2023-12-10T18:38:14.539 INFO:teuthology.orchestra.run.smithi076.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-12-10T18:38:14.539 INFO:teuthology.orchestra.run.smithi076.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-12-10T18:38:14.539 INFO:teuthology.orchestra.run.smithi076.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-12-10T18:38:14.539 INFO:teuthology.orchestra.run.smithi076.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-12-10T18:38:14.539 INFO:teuthology.orchestra.run.smithi076.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-12-10T18:38:14.539 INFO:teuthology.orchestra.run.smithi076.stdout:Use 'sudo apt autoremove' to remove them. 2023-12-10T18:38:14.576 INFO:teuthology.orchestra.run.smithi076.stdout:0 upgraded, 0 newly installed, 0 to remove and 275 not upgraded. 2023-12-10T18:38:14.578 DEBUG:teuthology.orchestra.run.smithi076:> dpkg -s linux-image-generic 2023-12-10T18:38:14.593 INFO:teuthology.orchestra.run.smithi076.stdout:Package: linux-image-generic 2023-12-10T18:38:14.594 INFO:teuthology.orchestra.run.smithi076.stdout:Status: install ok installed 2023-12-10T18:38:14.594 INFO:teuthology.orchestra.run.smithi076.stdout:Priority: optional 2023-12-10T18:38:14.594 INFO:teuthology.orchestra.run.smithi076.stdout:Section: kernel 2023-12-10T18:38:14.594 INFO:teuthology.orchestra.run.smithi076.stdout:Installed-Size: 20 2023-12-10T18:38:14.594 INFO:teuthology.orchestra.run.smithi076.stdout:Maintainer: Ubuntu Kernel Team 2023-12-10T18:38:14.594 INFO:teuthology.orchestra.run.smithi076.stdout:Architecture: amd64 2023-12-10T18:38:14.594 INFO:teuthology.orchestra.run.smithi076.stdout:Source: linux-meta 2023-12-10T18:38:14.595 INFO:teuthology.orchestra.run.smithi076.stdout:Version: 5.15.0.91.88 2023-12-10T18:38:14.595 INFO:teuthology.orchestra.run.smithi076.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-91), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-12-10T18:38:14.595 INFO:teuthology.orchestra.run.smithi076.stdout:Depends: linux-image-5.15.0-91-generic, linux-modules-extra-5.15.0-91-generic, linux-firmware, intel-microcode, amd64-microcode 2023-12-10T18:38:14.595 INFO:teuthology.orchestra.run.smithi076.stdout:Recommends: thermald 2023-12-10T18:38:14.595 INFO:teuthology.orchestra.run.smithi076.stdout:Description: Generic Linux kernel image 2023-12-10T18:38:14.595 INFO:teuthology.orchestra.run.smithi076.stdout: This package will always depend on the latest generic kernel image 2023-12-10T18:38:14.595 INFO:teuthology.orchestra.run.smithi076.stdout: available. 2023-12-10T18:38:14.595 DEBUG:teuthology.orchestra.run.smithi076:> mktemp 2023-12-10T18:38:14.639 INFO:teuthology.orchestra.run.smithi076.stdout:/tmp/tmp.iJX38R2a5q 2023-12-10T18:38:14.640 DEBUG:teuthology.orchestra.run.smithi076:> sudo cp /boot/grub/grub.cfg /tmp/tmp.iJX38R2a5q 2023-12-10T18:38:14.695 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod 0666 /tmp/tmp.iJX38R2a5q 2023-12-10T18:38:14.811 DEBUG:teuthology.orchestra.remote:smithi076:/tmp/tmp.iJX38R2a5q is 10KB 2023-12-10T18:38:14.824 DEBUG:teuthology.orchestra.run.smithi076:> rm -fr /tmp/tmp.iJX38R2a5q 2023-12-10T18:38:14.829 DEBUG:teuthology.orchestra.run.smithi076:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-12-10T18:38:14.883 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2023-12-10T18:38:14.883 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-12-10T18:38:14.883 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-12-10T18:38:14.951 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-91-generic 2023-12-10T18:38:14.951 DEBUG:teuthology.orchestra.run.smithi076:> sudo update-grub 2023-12-10T18:38:15.479 INFO:teuthology.orchestra.run.smithi033.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-12-10T18:38:15.479 INFO:teuthology.orchestra.run.smithi033.stdout:Sourcing file `/etc/default/grub' 2023-12-10T18:38:15.502 INFO:teuthology.orchestra.run.smithi033.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-12-10T18:38:15.525 INFO:teuthology.orchestra.run.smithi033.stdout:Generating grub configuration file ... 2023-12-10T18:38:15.808 INFO:teuthology.orchestra.run.smithi033.stdout:Found linux image: /boot/vmlinuz-5.15.0-91-generic 2023-12-10T18:38:15.835 INFO:teuthology.orchestra.run.smithi033.stdout:Found initrd image: /boot/initrd.img-5.15.0-91-generic 2023-12-10T18:38:16.150 INFO:teuthology.orchestra.run.smithi076.stderr:Sourcing file `/etc/default/grub' 2023-12-10T18:38:16.152 INFO:teuthology.orchestra.run.smithi076.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-12-10T18:38:16.153 INFO:teuthology.orchestra.run.smithi076.stderr:Generating grub configuration file ... 2023-12-10T18:38:16.254 INFO:teuthology.orchestra.run.smithi033.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-12-10T18:38:16.260 INFO:teuthology.orchestra.run.smithi033.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-12-10T18:38:16.334 INFO:teuthology.orchestra.run.smithi033.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-12-10T18:38:16.340 INFO:teuthology.orchestra.run.smithi033.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-12-10T18:38:16.357 INFO:teuthology.orchestra.run.smithi076.stderr:Found linux image: /boot/vmlinuz-5.15.0-91-generic 2023-12-10T18:38:16.363 INFO:teuthology.orchestra.run.smithi076.stderr:Found initrd image: /boot/initrd.img-5.15.0-91-generic 2023-12-10T18:38:16.511 INFO:teuthology.orchestra.run.smithi033.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2023-12-10T18:38:16.511 INFO:teuthology.orchestra.run.smithi033.stdout:Systems on them will not be added to the GRUB boot configuration. 2023-12-10T18:38:16.511 INFO:teuthology.orchestra.run.smithi033.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-12-10T18:38:16.554 INFO:teuthology.orchestra.run.smithi033.stdout:done 2023-12-10T18:38:16.583 INFO:teuthology.orchestra.run.smithi076.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-12-10T18:38:16.588 INFO:teuthology.orchestra.run.smithi076.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-12-10T18:38:16.632 INFO:teuthology.orchestra.run.smithi076.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-12-10T18:38:16.637 INFO:teuthology.orchestra.run.smithi076.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-12-10T18:38:16.757 INFO:teuthology.orchestra.run.smithi076.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2023-12-10T18:38:16.757 INFO:teuthology.orchestra.run.smithi076.stderr:Systems on them will not be added to the GRUB boot configuration. 2023-12-10T18:38:16.758 INFO:teuthology.orchestra.run.smithi076.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-12-10T18:38:16.778 INFO:teuthology.orchestra.run.smithi076.stderr:done 2023-12-10T18:38:16.780 DEBUG:teuthology.orchestra.run.smithi076:> sudo shutdown -r now 2023-12-10T18:38:17.010 DEBUG:teuthology.orchestra.run.smithi033:> dpkg -s linux-image-generic 2023-12-10T18:38:17.033 INFO:teuthology.orchestra.run.smithi033.stdout:Package: linux-image-generic 2023-12-10T18:38:17.033 INFO:teuthology.orchestra.run.smithi033.stdout:Status: install ok installed 2023-12-10T18:38:17.033 INFO:teuthology.orchestra.run.smithi033.stdout:Priority: optional 2023-12-10T18:38:17.034 INFO:teuthology.orchestra.run.smithi033.stdout:Section: kernel 2023-12-10T18:38:17.034 INFO:teuthology.orchestra.run.smithi033.stdout:Installed-Size: 20 2023-12-10T18:38:17.034 INFO:teuthology.orchestra.run.smithi033.stdout:Maintainer: Ubuntu Kernel Team 2023-12-10T18:38:17.034 INFO:teuthology.orchestra.run.smithi033.stdout:Architecture: amd64 2023-12-10T18:38:17.034 INFO:teuthology.orchestra.run.smithi033.stdout:Source: linux-meta 2023-12-10T18:38:17.034 INFO:teuthology.orchestra.run.smithi033.stdout:Version: 5.15.0.91.88 2023-12-10T18:38:17.034 INFO:teuthology.orchestra.run.smithi033.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-91), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-12-10T18:38:17.034 INFO:teuthology.orchestra.run.smithi033.stdout:Depends: linux-image-5.15.0-91-generic, linux-modules-extra-5.15.0-91-generic, linux-firmware, intel-microcode, amd64-microcode 2023-12-10T18:38:17.034 INFO:teuthology.orchestra.run.smithi033.stdout:Recommends: thermald 2023-12-10T18:38:17.034 INFO:teuthology.orchestra.run.smithi033.stdout:Description: Generic Linux kernel image 2023-12-10T18:38:17.034 INFO:teuthology.orchestra.run.smithi033.stdout: This package will always depend on the latest generic kernel image 2023-12-10T18:38:17.034 INFO:teuthology.orchestra.run.smithi033.stdout: available. 2023-12-10T18:38:17.035 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-91-generic 2023-12-10T18:38:17.035 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-10T18:38:17.035 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-10T18:38:17.035 INFO:teuthology.task.kernel:Installing distro kernel on cluster1.mon.a... 2023-12-10T18:38:17.035 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi033.front.sepia.ceph.com, path=None, version=distro) 2023-12-10T18:38:17.035 DEBUG:teuthology.orchestra.run.smithi033:> sudo apt-get clean 2023-12-10T18:38:17.353 DEBUG:teuthology.orchestra.run.smithi033:> sudo apt-get update 2023-12-10T18:38:17.623 INFO:teuthology.orchestra.run.smithi033.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-12-10T18:38:17.623 INFO:teuthology.orchestra.run.smithi033.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-12-10T18:38:17.687 INFO:teuthology.orchestra.run.smithi033.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-12-10T18:38:17.769 INFO:teuthology.orchestra.run.smithi033.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-12-10T18:38:18.980 INFO:teuthology.orchestra.run.smithi033.stdout:Reading package lists... 2023-12-10T18:38:18.997 DEBUG:teuthology.orchestra.run.smithi033:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-12-10T18:38:19.060 INFO:teuthology.orchestra.run.smithi033.stdout:Reading package lists... 2023-12-10T18:38:19.253 INFO:teuthology.orchestra.run.smithi033.stdout:Building dependency tree... 2023-12-10T18:38:19.254 INFO:teuthology.orchestra.run.smithi033.stdout:Reading state information... 2023-12-10T18:38:19.427 INFO:teuthology.orchestra.run.smithi033.stdout:linux-image-generic is already the newest version (5.15.0.91.88). 2023-12-10T18:38:19.427 INFO:teuthology.orchestra.run.smithi033.stdout:The following packages were automatically installed and are no longer required: 2023-12-10T18:38:19.427 INFO:teuthology.orchestra.run.smithi033.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-12-10T18:38:19.427 INFO:teuthology.orchestra.run.smithi033.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-12-10T18:38:19.427 INFO:teuthology.orchestra.run.smithi033.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-12-10T18:38:19.427 INFO:teuthology.orchestra.run.smithi033.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-12-10T18:38:19.428 INFO:teuthology.orchestra.run.smithi033.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-12-10T18:38:19.428 INFO:teuthology.orchestra.run.smithi033.stdout:Use 'sudo apt autoremove' to remove them. 2023-12-10T18:38:19.467 INFO:teuthology.orchestra.run.smithi033.stdout:0 upgraded, 0 newly installed, 0 to remove and 275 not upgraded. 2023-12-10T18:38:19.469 DEBUG:teuthology.orchestra.run.smithi033:> dpkg -s linux-image-generic 2023-12-10T18:38:19.486 INFO:teuthology.orchestra.run.smithi033.stdout:Package: linux-image-generic 2023-12-10T18:38:19.486 INFO:teuthology.orchestra.run.smithi033.stdout:Status: install ok installed 2023-12-10T18:38:19.486 INFO:teuthology.orchestra.run.smithi033.stdout:Priority: optional 2023-12-10T18:38:19.486 INFO:teuthology.orchestra.run.smithi033.stdout:Section: kernel 2023-12-10T18:38:19.486 INFO:teuthology.orchestra.run.smithi033.stdout:Installed-Size: 20 2023-12-10T18:38:19.487 INFO:teuthology.orchestra.run.smithi033.stdout:Maintainer: Ubuntu Kernel Team 2023-12-10T18:38:19.487 INFO:teuthology.orchestra.run.smithi033.stdout:Architecture: amd64 2023-12-10T18:38:19.487 INFO:teuthology.orchestra.run.smithi033.stdout:Source: linux-meta 2023-12-10T18:38:19.487 INFO:teuthology.orchestra.run.smithi033.stdout:Version: 5.15.0.91.88 2023-12-10T18:38:19.487 INFO:teuthology.orchestra.run.smithi033.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-91), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-12-10T18:38:19.487 INFO:teuthology.orchestra.run.smithi033.stdout:Depends: linux-image-5.15.0-91-generic, linux-modules-extra-5.15.0-91-generic, linux-firmware, intel-microcode, amd64-microcode 2023-12-10T18:38:19.487 INFO:teuthology.orchestra.run.smithi033.stdout:Recommends: thermald 2023-12-10T18:38:19.487 INFO:teuthology.orchestra.run.smithi033.stdout:Description: Generic Linux kernel image 2023-12-10T18:38:19.487 INFO:teuthology.orchestra.run.smithi033.stdout: This package will always depend on the latest generic kernel image 2023-12-10T18:38:19.487 INFO:teuthology.orchestra.run.smithi033.stdout: available. 2023-12-10T18:38:19.487 DEBUG:teuthology.orchestra.run.smithi033:> mktemp 2023-12-10T18:38:19.535 INFO:teuthology.orchestra.run.smithi033.stdout:/tmp/tmp.5e1Ybfwh7D 2023-12-10T18:38:19.535 DEBUG:teuthology.orchestra.run.smithi033:> sudo cp /boot/grub/grub.cfg /tmp/tmp.5e1Ybfwh7D 2023-12-10T18:38:19.590 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 0666 /tmp/tmp.5e1Ybfwh7D 2023-12-10T18:38:19.711 DEBUG:teuthology.orchestra.remote:smithi033:/tmp/tmp.5e1Ybfwh7D is 10KB 2023-12-10T18:38:19.725 DEBUG:teuthology.orchestra.run.smithi033:> rm -fr /tmp/tmp.5e1Ybfwh7D 2023-12-10T18:38:19.730 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-12-10T18:38:19.787 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-10T18:38:19.787 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-12-10T18:38:19.787 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-12-10T18:38:19.854 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-91-generic 2023-12-10T18:38:19.854 DEBUG:teuthology.orchestra.run.smithi033:> sudo update-grub 2023-12-10T18:38:21.062 INFO:teuthology.orchestra.run.smithi033.stderr:Sourcing file `/etc/default/grub' 2023-12-10T18:38:21.064 INFO:teuthology.orchestra.run.smithi033.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-12-10T18:38:21.067 INFO:teuthology.orchestra.run.smithi033.stderr:Generating grub configuration file ... 2023-12-10T18:38:21.220 INFO:teuthology.orchestra.run.smithi033.stderr:Found linux image: /boot/vmlinuz-5.15.0-91-generic 2023-12-10T18:38:21.226 INFO:teuthology.orchestra.run.smithi033.stderr:Found initrd image: /boot/initrd.img-5.15.0-91-generic 2023-12-10T18:38:21.464 INFO:teuthology.orchestra.run.smithi033.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-12-10T18:38:21.469 INFO:teuthology.orchestra.run.smithi033.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-12-10T18:38:21.512 INFO:teuthology.orchestra.run.smithi033.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-12-10T18:38:21.517 INFO:teuthology.orchestra.run.smithi033.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-12-10T18:38:21.653 INFO:teuthology.orchestra.run.smithi033.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2023-12-10T18:38:21.653 INFO:teuthology.orchestra.run.smithi033.stderr:Systems on them will not be added to the GRUB boot configuration. 2023-12-10T18:38:21.653 INFO:teuthology.orchestra.run.smithi033.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-12-10T18:38:21.669 INFO:teuthology.orchestra.run.smithi033.stderr:done 2023-12-10T18:38:21.679 DEBUG:teuthology.orchestra.run.smithi033:> sudo shutdown -r now 2023-12-10T18:38:46.806 INFO:teuthology.task.kernel:Checking client cluster2.mon.a for new kernel version... 2023-12-10T18:38:46.807 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi076.front.sepia.ceph.com' 2023-12-10T18:38:46.808 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-10T18:38:51.686 INFO:teuthology.task.kernel:Checking client cluster1.mon.a for new kernel version... 2023-12-10T18:38:51.686 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-12-10T18:38:51.687 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-10T18:39:05.314 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.76 2023-12-10T18:39:10.178 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.33 2023-12-10T18:39:14.319 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi076.front.sepia.ceph.com' 2023-12-10T18:39:14.320 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-10T18:39:17.382 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.76 2023-12-10T18:39:19.181 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-12-10T18:39:19.182 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-10T18:39:29.390 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi076.front.sepia.ceph.com' 2023-12-10T18:39:29.391 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-10T18:39:32.450 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.76 2023-12-10T18:39:37.574 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.33 2023-12-10T18:39:47.458 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi076.front.sepia.ceph.com' 2023-12-10T18:39:47.459 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-10T18:39:49.577 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-12-10T18:39:49.578 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-10T18:39:52.646 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.33 2023-12-10T18:40:05.990 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.76 2023-12-10T18:40:07.648 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-12-10T18:40:07.649 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-10T18:40:07.866 DEBUG:teuthology.orchestra.run.smithi033:> true 2023-12-10T18:40:08.497 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-12-10T18:40:08.497 INFO:teuthology.task.kernel:Checking kernel version of cluster1.mon.a, want "5.15.0-91-generic"... 2023-12-10T18:40:08.498 DEBUG:teuthology.orchestra.run.smithi033:> uname -r 2023-12-10T18:40:08.545 INFO:teuthology.orchestra.run.smithi033.stdout:5.15.0-91-generic 2023-12-10T18:40:08.545 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-91-generic vs 5.15.0-91-generic 2023-12-10T18:40:08.545 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-10T18:40:08.545 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-12-10T18:40:09.546 INFO:teuthology.task.kernel:Enabling kdb on cluster1.mon.a... 2023-12-10T18:40:09.546 DEBUG:teuthology.orchestra.run.smithi033:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-10T18:40:09.625 INFO:teuthology.orchestra.run.smithi033.stdout:ttyS1 2023-12-10T18:40:09.650 DEBUG:teuthology.parallel:result is None 2023-12-10T18:40:23.992 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi076.front.sepia.ceph.com' 2023-12-10T18:40:23.993 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-10T18:40:24.336 DEBUG:teuthology.orchestra.run.smithi076:> true 2023-12-10T18:40:24.961 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi076.front.sepia.ceph.com' 2023-12-10T18:40:24.961 INFO:teuthology.task.kernel:Checking kernel version of cluster2.mon.a, want "5.15.0-91-generic"... 2023-12-10T18:40:24.961 DEBUG:teuthology.orchestra.run.smithi076:> uname -r 2023-12-10T18:40:25.008 INFO:teuthology.orchestra.run.smithi076.stdout:5.15.0-91-generic 2023-12-10T18:40:25.008 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-91-generic vs 5.15.0-91-generic 2023-12-10T18:40:25.008 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-10T18:40:25.008 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-12-10T18:40:26.009 INFO:teuthology.task.kernel:Enabling kdb on cluster2.mon.a... 2023-12-10T18:40:26.010 DEBUG:teuthology.orchestra.run.smithi076:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-10T18:40:26.353 INFO:teuthology.orchestra.run.smithi076.stdout:ttyS1 2023-12-10T18:40:26.382 DEBUG:teuthology.parallel:result is None 2023-12-10T18:40:26.383 INFO:teuthology.run_tasks:Running task internal.base... 2023-12-10T18:40:26.390 INFO:teuthology.task.internal:Creating test directory... 2023-12-10T18:40:26.390 DEBUG:teuthology.orchestra.run.smithi033:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-10T18:40:26.392 DEBUG:teuthology.orchestra.run.smithi076:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-10T18:40:26.398 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-12-10T18:40:26.405 INFO:teuthology.run_tasks:Running task internal.archive... 2023-12-10T18:40:26.412 INFO:teuthology.task.internal:Creating archive directory... 2023-12-10T18:40:26.412 DEBUG:teuthology.orchestra.run.smithi033:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-10T18:40:26.442 DEBUG:teuthology.orchestra.run.smithi076:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-10T18:40:26.471 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-12-10T18:40:26.478 INFO:teuthology.task.internal:Enabling coredump saving... 2023-12-10T18:40:26.478 DEBUG:teuthology.orchestra.run.smithi033:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-10T18:40:26.493 DEBUG:teuthology.orchestra.run.smithi076:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-10T18:40:26.512 INFO:teuthology.orchestra.run.smithi033.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-10T18:40:26.521 INFO:teuthology.orchestra.run.smithi033.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-10T18:40:26.531 INFO:teuthology.orchestra.run.smithi076.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-10T18:40:26.541 INFO:teuthology.orchestra.run.smithi076.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-10T18:40:26.542 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-12-10T18:40:26.549 INFO:teuthology.task.internal:Configuring sudo... 2023-12-10T18:40:26.549 DEBUG:teuthology.orchestra.run.smithi033:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-10T18:40:26.565 DEBUG:teuthology.orchestra.run.smithi076:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-10T18:40:26.599 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-12-10T18:40:26.609 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-12-10T18:40:26.609 DEBUG:teuthology.orchestra.run.smithi033:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-10T18:40:26.621 DEBUG:teuthology.orchestra.run.smithi076:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-10T18:40:26.648 DEBUG:teuthology.orchestra.run.smithi033:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-10T18:40:26.672 DEBUG:teuthology.orchestra.run.smithi033:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-10T18:40:26.721 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-10T18:40:26.721 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-10T18:40:26.783 DEBUG:teuthology.orchestra.run.smithi076:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-10T18:40:26.788 DEBUG:teuthology.orchestra.run.smithi076:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-10T18:40:26.836 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2023-12-10T18:40:26.836 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-10T18:40:26.906 DEBUG:teuthology.orchestra.run.smithi033:> sudo service rsyslog restart 2023-12-10T18:40:26.909 DEBUG:teuthology.orchestra.run.smithi076:> sudo service rsyslog restart 2023-12-10T18:40:26.984 INFO:teuthology.run_tasks:Running task internal.timer... 2023-12-10T18:40:26.991 INFO:teuthology.task.internal:Starting timer... 2023-12-10T18:40:26.992 INFO:teuthology.run_tasks:Running task pcp... 2023-12-10T18:40:27.001 INFO:teuthology.run_tasks:Running task selinux... 2023-12-10T18:40:27.010 DEBUG:teuthology.task.selinux:Excluding smithi033: OS 'ubuntu' does not support SELinux 2023-12-10T18:40:27.010 DEBUG:teuthology.task.selinux:Excluding smithi076: OS 'ubuntu' does not support SELinux 2023-12-10T18:40:27.011 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-12-10T18:40:27.011 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2023-12-10T18:40:27.011 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-12-10T18:40:27.011 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-12-10T18:40:27.020 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-12-10T18:40:27.025 INFO:teuthology.repo_utils:Fetching main from origin 2023-12-10T18:40:27.233 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-12-10T18:40:27.249 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-12-10T18:40:27.250 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi033.front.sepia.ceph.com,smithi076.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-12-10T18:48:15.318 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi033.front.sepia.ceph.com'), Remote(name='ubuntu@smithi076.front.sepia.ceph.com')] 2023-12-10T18:48:15.320 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-12-10T18:48:15.321 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-10T18:48:15.442 DEBUG:teuthology.orchestra.run.smithi033:> true 2023-12-10T18:48:15.522 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-12-10T18:48:15.523 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi076.front.sepia.ceph.com' 2023-12-10T18:48:15.523 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-10T18:48:15.598 DEBUG:teuthology.orchestra.run.smithi076:> true 2023-12-10T18:48:15.673 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi076.front.sepia.ceph.com' 2023-12-10T18:48:15.674 INFO:teuthology.run_tasks:Running task clock... 2023-12-10T18:48:15.686 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-12-10T18:48:15.686 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-10T18:48:15.686 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-10T18:48:15.689 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-10T18:48:15.689 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-10T18:48:15.720 INFO:teuthology.orchestra.run.smithi033.stdout:10 Dec 18:48:15 ntpd[27879]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2023-12-10T18:48:15.720 INFO:teuthology.orchestra.run.smithi033.stdout:10 Dec 18:48:15 ntpd[27879]: Command line: ntpd -gq 2023-12-10T18:48:15.720 INFO:teuthology.orchestra.run.smithi033.stdout:10 Dec 18:48:15 ntpd[27879]: ---------------------------------------------------- 2023-12-10T18:48:15.720 INFO:teuthology.orchestra.run.smithi033.stdout:10 Dec 18:48:15 ntpd[27879]: ntp-4 is maintained by Network Time Foundation, 2023-12-10T18:48:15.720 INFO:teuthology.orchestra.run.smithi033.stdout:10 Dec 18:48:15 ntpd[27879]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2023-12-10T18:48:15.720 INFO:teuthology.orchestra.run.smithi033.stdout:10 Dec 18:48:15 ntpd[27879]: corporation. Support and training for ntp-4 are 2023-12-10T18:48:15.720 INFO:teuthology.orchestra.run.smithi033.stdout:10 Dec 18:48:15 ntpd[27879]: available at https://www.nwtime.org/support 2023-12-10T18:48:15.720 INFO:teuthology.orchestra.run.smithi033.stdout:10 Dec 18:48:15 ntpd[27879]: ---------------------------------------------------- 2023-12-10T18:48:15.721 INFO:teuthology.orchestra.run.smithi033.stdout:10 Dec 18:48:15 ntpd[27879]: proto: precision = 0.063 usec (-24) 2023-12-10T18:48:15.721 INFO:teuthology.orchestra.run.smithi033.stdout:10 Dec 18:48:15 ntpd[27879]: basedate set to 2022-02-04 2023-12-10T18:48:15.721 INFO:teuthology.orchestra.run.smithi033.stdout:10 Dec 18:48:15 ntpd[27879]: gps base set to 2022-02-06 (week 2196) 2023-12-10T18:48:15.721 INFO:teuthology.orchestra.run.smithi033.stdout:10 Dec 18:48:15 ntpd[27879]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-12-10T18:48:15.722 INFO:teuthology.orchestra.run.smithi033.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-12-10T18:48:15.722 INFO:teuthology.orchestra.run.smithi033.stderr:restrict ::: KOD does nothing without LIMITED. 2023-12-10T18:48:15.723 INFO:teuthology.orchestra.run.smithi033.stdout:10 Dec 18:48:15 ntpd[27879]: restrict ::: KOD does nothing without LIMITED. 2023-12-10T18:48:15.723 INFO:teuthology.orchestra.run.smithi033.stdout:10 Dec 18:48:15 ntpd[27879]: Listen and drop on 0 v6wildcard [::]:123 2023-12-10T18:48:15.723 INFO:teuthology.orchestra.run.smithi033.stdout:10 Dec 18:48:15 ntpd[27879]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-12-10T18:48:15.723 INFO:teuthology.orchestra.run.smithi033.stdout:10 Dec 18:48:15 ntpd[27879]: Listen normally on 2 lo 127.0.0.1:123 2023-12-10T18:48:15.723 INFO:teuthology.orchestra.run.smithi033.stdout:10 Dec 18:48:15 ntpd[27879]: Listen normally on 3 ens1f0 172.21.15.33:123 2023-12-10T18:48:15.723 INFO:teuthology.orchestra.run.smithi033.stdout:10 Dec 18:48:15 ntpd[27879]: Listen normally on 4 lo [::1]:123 2023-12-10T18:48:15.723 INFO:teuthology.orchestra.run.smithi033.stdout:10 Dec 18:48:15 ntpd[27879]: Listen normally on 5 ens1f0 [fe80::ae1f:6bff:fea5:d294%2]:123 2023-12-10T18:48:15.723 INFO:teuthology.orchestra.run.smithi033.stdout:10 Dec 18:48:15 ntpd[27879]: Listening on routing socket on fd #22 for interface updates 2023-12-10T18:48:15.742 INFO:teuthology.orchestra.run.smithi076.stdout:10 Dec 18:48:15 ntpd[27976]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2023-12-10T18:48:15.742 INFO:teuthology.orchestra.run.smithi076.stdout:10 Dec 18:48:15 ntpd[27976]: Command line: ntpd -gq 2023-12-10T18:48:15.742 INFO:teuthology.orchestra.run.smithi076.stdout:10 Dec 18:48:15 ntpd[27976]: ---------------------------------------------------- 2023-12-10T18:48:15.742 INFO:teuthology.orchestra.run.smithi076.stdout:10 Dec 18:48:15 ntpd[27976]: ntp-4 is maintained by Network Time Foundation, 2023-12-10T18:48:15.742 INFO:teuthology.orchestra.run.smithi076.stdout:10 Dec 18:48:15 ntpd[27976]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2023-12-10T18:48:15.742 INFO:teuthology.orchestra.run.smithi076.stdout:10 Dec 18:48:15 ntpd[27976]: corporation. Support and training for ntp-4 are 2023-12-10T18:48:15.742 INFO:teuthology.orchestra.run.smithi076.stdout:10 Dec 18:48:15 ntpd[27976]: available at https://www.nwtime.org/support 2023-12-10T18:48:15.742 INFO:teuthology.orchestra.run.smithi076.stdout:10 Dec 18:48:15 ntpd[27976]: ---------------------------------------------------- 2023-12-10T18:48:15.742 INFO:teuthology.orchestra.run.smithi076.stdout:10 Dec 18:48:15 ntpd[27976]: proto: precision = 0.055 usec (-24) 2023-12-10T18:48:15.745 INFO:teuthology.orchestra.run.smithi076.stdout:10 Dec 18:48:15 ntpd[27976]: basedate set to 2022-02-04 2023-12-10T18:48:15.745 INFO:teuthology.orchestra.run.smithi076.stdout:10 Dec 18:48:15 ntpd[27976]: gps base set to 2022-02-06 (week 2196) 2023-12-10T18:48:15.746 INFO:teuthology.orchestra.run.smithi076.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-12-10T18:48:15.746 INFO:teuthology.orchestra.run.smithi076.stdout:10 Dec 18:48:15 ntpd[27976]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-12-10T18:48:15.746 INFO:teuthology.orchestra.run.smithi076.stdout:10 Dec 18:48:15 ntpd[27976]: restrict ::: KOD does nothing without LIMITED. 2023-12-10T18:48:15.746 INFO:teuthology.orchestra.run.smithi076.stderr:restrict ::: KOD does nothing without LIMITED. 2023-12-10T18:48:15.747 INFO:teuthology.orchestra.run.smithi076.stdout:10 Dec 18:48:15 ntpd[27976]: Listen and drop on 0 v6wildcard [::]:123 2023-12-10T18:48:15.747 INFO:teuthology.orchestra.run.smithi076.stdout:10 Dec 18:48:15 ntpd[27976]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-12-10T18:48:15.747 INFO:teuthology.orchestra.run.smithi076.stdout:10 Dec 18:48:15 ntpd[27976]: Listen normally on 2 lo 127.0.0.1:123 2023-12-10T18:48:15.747 INFO:teuthology.orchestra.run.smithi076.stdout:10 Dec 18:48:15 ntpd[27976]: Listen normally on 3 enp3s0f1 172.21.15.76:123 2023-12-10T18:48:15.748 INFO:teuthology.orchestra.run.smithi076.stdout:10 Dec 18:48:15 ntpd[27976]: Listen normally on 4 lo [::1]:123 2023-12-10T18:48:15.748 INFO:teuthology.orchestra.run.smithi076.stdout:10 Dec 18:48:15 ntpd[27976]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:47e1%5]:123 2023-12-10T18:48:15.748 INFO:teuthology.orchestra.run.smithi076.stdout:10 Dec 18:48:15 ntpd[27976]: Listening on routing socket on fd #22 for interface updates 2023-12-10T18:48:16.720 INFO:teuthology.orchestra.run.smithi033.stderr:10 Dec 18:48:16 ntpd[27879]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-10T18:48:16.721 INFO:teuthology.orchestra.run.smithi033.stderr:10 Dec 18:48:16 ntpd[27879]: can't open /var/log/ntpstats/rawstats.20231210: Permission denied 2023-12-10T18:48:16.721 INFO:teuthology.orchestra.run.smithi033.stderr:10 Dec 18:48:16 ntpd[27879]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-10T18:48:16.721 INFO:teuthology.orchestra.run.smithi033.stderr:10 Dec 18:48:16 ntpd[27879]: can't open /var/log/ntpstats/rawstats.20231210: Permission denied 2023-12-10T18:48:16.721 INFO:teuthology.orchestra.run.smithi033.stderr:10 Dec 18:48:16 ntpd[27879]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-10T18:48:16.721 INFO:teuthology.orchestra.run.smithi033.stderr:10 Dec 18:48:16 ntpd[27879]: can't open /var/log/ntpstats/peerstats.20231210: Permission denied 2023-12-10T18:48:16.742 INFO:teuthology.orchestra.run.smithi076.stderr:10 Dec 18:48:16 ntpd[27976]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-10T18:48:16.742 INFO:teuthology.orchestra.run.smithi076.stderr:10 Dec 18:48:16 ntpd[27976]: can't open /var/log/ntpstats/rawstats.20231210: Permission denied 2023-12-10T18:48:16.742 INFO:teuthology.orchestra.run.smithi076.stderr:10 Dec 18:48:16 ntpd[27976]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-10T18:48:16.742 INFO:teuthology.orchestra.run.smithi076.stderr:10 Dec 18:48:16 ntpd[27976]: can't open /var/log/ntpstats/rawstats.20231210: Permission denied 2023-12-10T18:48:16.742 INFO:teuthology.orchestra.run.smithi076.stderr:10 Dec 18:48:16 ntpd[27976]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-10T18:48:16.743 INFO:teuthology.orchestra.run.smithi076.stderr:10 Dec 18:48:16 ntpd[27976]: can't open /var/log/ntpstats/rawstats.20231210: Permission denied 2023-12-10T18:48:16.743 INFO:teuthology.orchestra.run.smithi076.stderr:10 Dec 18:48:16 ntpd[27976]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-10T18:48:16.743 INFO:teuthology.orchestra.run.smithi076.stderr:10 Dec 18:48:16 ntpd[27976]: can't open /var/log/ntpstats/peerstats.20231210: Permission denied 2023-12-10T18:48:18.720 INFO:teuthology.orchestra.run.smithi033.stderr:10 Dec 18:48:18 ntpd[27879]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-10T18:48:18.721 INFO:teuthology.orchestra.run.smithi033.stderr:10 Dec 18:48:18 ntpd[27879]: can't open /var/log/ntpstats/rawstats.20231210: Permission denied 2023-12-10T18:48:18.721 INFO:teuthology.orchestra.run.smithi033.stderr:10 Dec 18:48:18 ntpd[27879]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-10T18:48:18.721 INFO:teuthology.orchestra.run.smithi033.stderr:10 Dec 18:48:18 ntpd[27879]: can't open /var/log/ntpstats/rawstats.20231210: Permission denied 2023-12-10T18:48:18.721 INFO:teuthology.orchestra.run.smithi033.stderr:10 Dec 18:48:18 ntpd[27879]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-10T18:48:18.721 INFO:teuthology.orchestra.run.smithi033.stderr:10 Dec 18:48:18 ntpd[27879]: can't open /var/log/ntpstats/peerstats.20231210: Permission denied 2023-12-10T18:48:18.742 INFO:teuthology.orchestra.run.smithi076.stderr:10 Dec 18:48:18 ntpd[27976]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-10T18:48:18.743 INFO:teuthology.orchestra.run.smithi076.stderr:10 Dec 18:48:18 ntpd[27976]: can't open /var/log/ntpstats/rawstats.20231210: Permission denied 2023-12-10T18:48:18.743 INFO:teuthology.orchestra.run.smithi076.stderr:10 Dec 18:48:18 ntpd[27976]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-10T18:48:18.743 INFO:teuthology.orchestra.run.smithi076.stderr:10 Dec 18:48:18 ntpd[27976]: can't open /var/log/ntpstats/rawstats.20231210: Permission denied 2023-12-10T18:48:18.743 INFO:teuthology.orchestra.run.smithi076.stderr:10 Dec 18:48:18 ntpd[27976]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-10T18:48:18.743 INFO:teuthology.orchestra.run.smithi076.stderr:10 Dec 18:48:18 ntpd[27976]: can't open /var/log/ntpstats/rawstats.20231210: Permission denied 2023-12-10T18:48:18.743 INFO:teuthology.orchestra.run.smithi076.stderr:10 Dec 18:48:18 ntpd[27976]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-10T18:48:18.743 INFO:teuthology.orchestra.run.smithi076.stderr:10 Dec 18:48:18 ntpd[27976]: can't open /var/log/ntpstats/peerstats.20231210: Permission denied 2023-12-10T18:48:18.743 INFO:teuthology.orchestra.run.smithi076.stderr:10 Dec 18:48:18 ntpd[27976]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-10T18:48:18.743 INFO:teuthology.orchestra.run.smithi076.stderr:10 Dec 18:48:18 ntpd[27976]: can't open /var/log/ntpstats/rawstats.20231210: Permission denied 2023-12-10T18:48:18.743 INFO:teuthology.orchestra.run.smithi076.stderr:10 Dec 18:48:18 ntpd[27976]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-10T18:48:18.743 INFO:teuthology.orchestra.run.smithi076.stderr:10 Dec 18:48:18 ntpd[27976]: can't open /var/log/ntpstats/peerstats.20231210: Permission denied 2023-12-10T18:48:19.721 INFO:teuthology.orchestra.run.smithi033.stderr:10 Dec 18:48:19 ntpd[27879]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-10T18:48:19.721 INFO:teuthology.orchestra.run.smithi033.stderr:10 Dec 18:48:19 ntpd[27879]: can't open /var/log/ntpstats/rawstats.20231210: Permission denied 2023-12-10T18:48:19.721 INFO:teuthology.orchestra.run.smithi033.stderr:10 Dec 18:48:19 ntpd[27879]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-10T18:48:19.721 INFO:teuthology.orchestra.run.smithi033.stderr:10 Dec 18:48:19 ntpd[27879]: can't open /var/log/ntpstats/rawstats.20231210: Permission denied 2023-12-10T18:48:19.721 INFO:teuthology.orchestra.run.smithi033.stderr:10 Dec 18:48:19 ntpd[27879]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-10T18:48:19.721 INFO:teuthology.orchestra.run.smithi033.stderr:10 Dec 18:48:19 ntpd[27879]: can't open /var/log/ntpstats/rawstats.20231210: Permission denied 2023-12-10T18:48:19.721 INFO:teuthology.orchestra.run.smithi033.stderr:10 Dec 18:48:19 ntpd[27879]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-10T18:48:19.721 INFO:teuthology.orchestra.run.smithi033.stderr:10 Dec 18:48:19 ntpd[27879]: can't open /var/log/ntpstats/peerstats.20231210: Permission denied 2023-12-10T18:48:20.722 INFO:teuthology.orchestra.run.smithi033.stderr:10 Dec 18:48:20 ntpd[27879]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-10T18:48:20.722 INFO:teuthology.orchestra.run.smithi033.stderr:10 Dec 18:48:20 ntpd[27879]: can't open /var/log/ntpstats/rawstats.20231210: Permission denied 2023-12-10T18:48:20.722 INFO:teuthology.orchestra.run.smithi033.stderr:10 Dec 18:48:20 ntpd[27879]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-10T18:48:20.723 INFO:teuthology.orchestra.run.smithi033.stderr:10 Dec 18:48:20 ntpd[27879]: can't open /var/log/ntpstats/rawstats.20231210: Permission denied 2023-12-10T18:48:20.723 INFO:teuthology.orchestra.run.smithi033.stderr:10 Dec 18:48:20 ntpd[27879]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-10T18:48:20.723 INFO:teuthology.orchestra.run.smithi033.stderr:10 Dec 18:48:20 ntpd[27879]: can't open /var/log/ntpstats/rawstats.20231210: Permission denied 2023-12-10T18:48:20.723 INFO:teuthology.orchestra.run.smithi033.stderr:10 Dec 18:48:20 ntpd[27879]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-10T18:48:20.723 INFO:teuthology.orchestra.run.smithi033.stderr:10 Dec 18:48:20 ntpd[27879]: can't open /var/log/ntpstats/peerstats.20231210: Permission denied 2023-12-10T18:48:20.723 INFO:teuthology.orchestra.run.smithi033.stderr:10 Dec 18:48:20 ntpd[27879]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-10T18:48:20.723 INFO:teuthology.orchestra.run.smithi033.stderr:10 Dec 18:48:20 ntpd[27879]: can't open /var/log/ntpstats/rawstats.20231210: Permission denied 2023-12-10T18:48:20.723 INFO:teuthology.orchestra.run.smithi033.stderr:10 Dec 18:48:20 ntpd[27879]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-10T18:48:20.724 INFO:teuthology.orchestra.run.smithi033.stderr:10 Dec 18:48:20 ntpd[27879]: can't open /var/log/ntpstats/peerstats.20231210: Permission denied 2023-12-10T18:48:20.744 INFO:teuthology.orchestra.run.smithi076.stderr:10 Dec 18:48:20 ntpd[27976]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-10T18:48:20.744 INFO:teuthology.orchestra.run.smithi076.stderr:10 Dec 18:48:20 ntpd[27976]: can't open /var/log/ntpstats/rawstats.20231210: Permission denied 2023-12-10T18:48:20.744 INFO:teuthology.orchestra.run.smithi076.stderr:10 Dec 18:48:20 ntpd[27976]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-10T18:48:20.744 INFO:teuthology.orchestra.run.smithi076.stderr:10 Dec 18:48:20 ntpd[27976]: can't open /var/log/ntpstats/rawstats.20231210: Permission denied 2023-12-10T18:48:20.744 INFO:teuthology.orchestra.run.smithi076.stderr:10 Dec 18:48:20 ntpd[27976]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-10T18:48:20.744 INFO:teuthology.orchestra.run.smithi076.stderr:10 Dec 18:48:20 ntpd[27976]: can't open /var/log/ntpstats/rawstats.20231210: Permission denied 2023-12-10T18:48:20.744 INFO:teuthology.orchestra.run.smithi076.stderr:10 Dec 18:48:20 ntpd[27976]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-10T18:48:20.744 INFO:teuthology.orchestra.run.smithi076.stderr:10 Dec 18:48:20 ntpd[27976]: can't open /var/log/ntpstats/rawstats.20231210: Permission denied 2023-12-10T18:48:20.745 INFO:teuthology.orchestra.run.smithi076.stderr:10 Dec 18:48:20 ntpd[27976]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-10T18:48:20.745 INFO:teuthology.orchestra.run.smithi076.stderr:10 Dec 18:48:20 ntpd[27976]: can't open /var/log/ntpstats/peerstats.20231210: Permission denied 2023-12-10T18:48:20.745 INFO:teuthology.orchestra.run.smithi076.stderr:10 Dec 18:48:20 ntpd[27976]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-10T18:48:20.745 INFO:teuthology.orchestra.run.smithi076.stderr:10 Dec 18:48:20 ntpd[27976]: can't open /var/log/ntpstats/rawstats.20231210: Permission denied 2023-12-10T18:48:20.745 INFO:teuthology.orchestra.run.smithi076.stderr:10 Dec 18:48:20 ntpd[27976]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-10T18:48:20.745 INFO:teuthology.orchestra.run.smithi076.stderr:10 Dec 18:48:20 ntpd[27976]: can't open /var/log/ntpstats/peerstats.20231210: Permission denied 2023-12-10T18:48:20.745 INFO:teuthology.orchestra.run.smithi076.stderr:10 Dec 18:48:20 ntpd[27976]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-10T18:48:20.745 INFO:teuthology.orchestra.run.smithi076.stderr:10 Dec 18:48:20 ntpd[27976]: can't open /var/log/ntpstats/rawstats.20231210: Permission denied 2023-12-10T18:48:20.745 INFO:teuthology.orchestra.run.smithi076.stderr:10 Dec 18:48:20 ntpd[27976]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-10T18:48:20.745 INFO:teuthology.orchestra.run.smithi076.stderr:10 Dec 18:48:20 ntpd[27976]: can't open /var/log/ntpstats/peerstats.20231210: Permission denied 2023-12-10T18:48:21.720 INFO:teuthology.orchestra.run.smithi033.stderr:10 Dec 18:48:21 ntpd[27879]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-10T18:48:21.720 INFO:teuthology.orchestra.run.smithi033.stderr:10 Dec 18:48:21 ntpd[27879]: can't open /var/log/ntpstats/rawstats.20231210: Permission denied 2023-12-10T18:48:21.721 INFO:teuthology.orchestra.run.smithi033.stderr:10 Dec 18:48:21 ntpd[27879]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-10T18:48:21.721 INFO:teuthology.orchestra.run.smithi033.stderr:10 Dec 18:48:21 ntpd[27879]: can't open /var/log/ntpstats/rawstats.20231210: Permission denied 2023-12-10T18:48:21.721 INFO:teuthology.orchestra.run.smithi033.stderr:10 Dec 18:48:21 ntpd[27879]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-10T18:48:21.721 INFO:teuthology.orchestra.run.smithi033.stderr:10 Dec 18:48:21 ntpd[27879]: can't open /var/log/ntpstats/peerstats.20231210: Permission denied 2023-12-10T18:48:22.721 INFO:teuthology.orchestra.run.smithi033.stderr:10 Dec 18:48:22 ntpd[27879]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-10T18:48:22.721 INFO:teuthology.orchestra.run.smithi033.stderr:10 Dec 18:48:22 ntpd[27879]: can't open /var/log/ntpstats/rawstats.20231210: Permission denied 2023-12-10T18:48:22.721 INFO:teuthology.orchestra.run.smithi033.stderr:10 Dec 18:48:22 ntpd[27879]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-10T18:48:22.721 INFO:teuthology.orchestra.run.smithi033.stderr:10 Dec 18:48:22 ntpd[27879]: can't open /var/log/ntpstats/rawstats.20231210: Permission denied 2023-12-10T18:48:22.721 INFO:teuthology.orchestra.run.smithi033.stderr:10 Dec 18:48:22 ntpd[27879]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-10T18:48:22.721 INFO:teuthology.orchestra.run.smithi033.stderr:10 Dec 18:48:22 ntpd[27879]: can't open /var/log/ntpstats/rawstats.20231210: Permission denied 2023-12-10T18:48:22.721 INFO:teuthology.orchestra.run.smithi033.stderr:10 Dec 18:48:22 ntpd[27879]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-10T18:48:22.721 INFO:teuthology.orchestra.run.smithi033.stderr:10 Dec 18:48:22 ntpd[27879]: can't open /var/log/ntpstats/peerstats.20231210: Permission denied 2023-12-10T18:48:22.721 INFO:teuthology.orchestra.run.smithi033.stdout:10 Dec 18:48:22 ntpd[27879]: ntpd: time slew +0.003898 s 2023-12-10T18:48:22.721 INFO:teuthology.orchestra.run.smithi033.stdout:ntpd: time slew +0.003898s 2023-12-10T18:48:22.722 INFO:teuthology.orchestra.run.smithi033.stderr:10 Dec 18:48:22 ntpd[27879]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-12-10T18:48:22.722 INFO:teuthology.orchestra.run.smithi033.stderr:10 Dec 18:48:22 ntpd[27879]: can't open /var/log/ntpstats/loopstats.20231210: Permission denied 2023-12-10T18:48:22.743 INFO:teuthology.orchestra.run.smithi076.stderr:10 Dec 18:48:22 ntpd[27976]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-10T18:48:22.743 INFO:teuthology.orchestra.run.smithi076.stderr:10 Dec 18:48:22 ntpd[27976]: can't open /var/log/ntpstats/rawstats.20231210: Permission denied 2023-12-10T18:48:22.743 INFO:teuthology.orchestra.run.smithi076.stderr:10 Dec 18:48:22 ntpd[27976]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-10T18:48:22.743 INFO:teuthology.orchestra.run.smithi076.stderr:10 Dec 18:48:22 ntpd[27976]: can't open /var/log/ntpstats/rawstats.20231210: Permission denied 2023-12-10T18:48:22.743 INFO:teuthology.orchestra.run.smithi076.stderr:10 Dec 18:48:22 ntpd[27976]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-10T18:48:22.743 INFO:teuthology.orchestra.run.smithi076.stderr:10 Dec 18:48:22 ntpd[27976]: can't open /var/log/ntpstats/rawstats.20231210: Permission denied 2023-12-10T18:48:22.744 INFO:teuthology.orchestra.run.smithi076.stderr:10 Dec 18:48:22 ntpd[27976]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-10T18:48:22.744 INFO:teuthology.orchestra.run.smithi076.stderr:10 Dec 18:48:22 ntpd[27976]: can't open /var/log/ntpstats/rawstats.20231210: Permission denied 2023-12-10T18:48:22.744 INFO:teuthology.orchestra.run.smithi076.stderr:10 Dec 18:48:22 ntpd[27976]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-10T18:48:22.744 INFO:teuthology.orchestra.run.smithi076.stderr:10 Dec 18:48:22 ntpd[27976]: can't open /var/log/ntpstats/peerstats.20231210: Permission denied 2023-12-10T18:48:22.744 INFO:teuthology.orchestra.run.smithi076.stderr:10 Dec 18:48:22 ntpd[27976]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-10T18:48:22.744 INFO:teuthology.orchestra.run.smithi076.stderr:10 Dec 18:48:22 ntpd[27976]: can't open /var/log/ntpstats/rawstats.20231210: Permission denied 2023-12-10T18:48:22.744 INFO:teuthology.orchestra.run.smithi076.stderr:10 Dec 18:48:22 ntpd[27976]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-10T18:48:22.744 INFO:teuthology.orchestra.run.smithi076.stderr:10 Dec 18:48:22 ntpd[27976]: can't open /var/log/ntpstats/peerstats.20231210: Permission denied 2023-12-10T18:48:22.744 INFO:teuthology.orchestra.run.smithi076.stderr:10 Dec 18:48:22 ntpd[27976]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-10T18:48:22.744 INFO:teuthology.orchestra.run.smithi076.stderr:10 Dec 18:48:22 ntpd[27976]: can't open /var/log/ntpstats/rawstats.20231210: Permission denied 2023-12-10T18:48:22.744 INFO:teuthology.orchestra.run.smithi076.stderr:10 Dec 18:48:22 ntpd[27976]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-10T18:48:22.744 INFO:teuthology.orchestra.run.smithi076.stderr:10 Dec 18:48:22 ntpd[27976]: can't open /var/log/ntpstats/peerstats.20231210: Permission denied 2023-12-10T18:48:22.744 INFO:teuthology.orchestra.run.smithi076.stdout:10 Dec 18:48:22 ntpd[27976]: ntpd: time slew -0.004973 s 2023-12-10T18:48:22.745 INFO:teuthology.orchestra.run.smithi076.stdout:ntpd: time slew -0.004973s 2023-12-10T18:48:22.745 INFO:teuthology.orchestra.run.smithi076.stderr:10 Dec 18:48:22 ntpd[27976]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-12-10T18:48:22.745 INFO:teuthology.orchestra.run.smithi076.stderr:10 Dec 18:48:22 ntpd[27976]: can't open /var/log/ntpstats/loopstats.20231210: Permission denied 2023-12-10T18:48:22.770 INFO:teuthology.orchestra.run.smithi033.stdout: remote refid st t when poll reach delay offset jitter 2023-12-10T18:48:22.770 INFO:teuthology.orchestra.run.smithi033.stdout:============================================================================== 2023-12-10T18:48:22.771 INFO:teuthology.orchestra.run.smithi033.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-12-10T18:48:22.771 INFO:teuthology.orchestra.run.smithi033.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-12-10T18:48:22.771 INFO:teuthology.orchestra.run.smithi033.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-12-10T18:48:22.771 INFO:teuthology.orchestra.run.smithi033.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-12-10T18:48:22.786 INFO:teuthology.orchestra.run.smithi076.stdout: remote refid st t when poll reach delay offset jitter 2023-12-10T18:48:22.787 INFO:teuthology.orchestra.run.smithi076.stdout:============================================================================== 2023-12-10T18:48:22.787 INFO:teuthology.orchestra.run.smithi076.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-12-10T18:48:22.787 INFO:teuthology.orchestra.run.smithi076.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-12-10T18:48:22.787 INFO:teuthology.orchestra.run.smithi076.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-12-10T18:48:22.787 INFO:teuthology.orchestra.run.smithi076.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-12-10T18:48:22.788 INFO:teuthology.run_tasks:Running task install... 2023-12-10T18:48:22.797 DEBUG:teuthology.task.install:project ceph 2023-12-10T18:48:22.797 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '3d2db44fa991bc355074aa9b7647f63b77201fef'}} 2023-12-10T18:48:22.797 DEBUG:teuthology.task.install:config {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': '3d2db44fa991bc355074aa9b7647f63b77201fef'} 2023-12-10T18:48:22.797 INFO:teuthology.task.install:Using flavor: default 2023-12-10T18:48:22.803 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-12-10T18:48:22.803 INFO:teuthology.task.install:extra packages: ['rbd-mirror'] 2023-12-10T18:48:22.804 DEBUG:teuthology.orchestra.run.smithi033:> sudo apt-key list | grep Ceph 2023-12-10T18:48:22.804 DEBUG:teuthology.orchestra.run.smithi076:> sudo apt-key list | grep Ceph 2023-12-10T18:48:22.872 INFO:teuthology.orchestra.run.smithi033.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-12-10T18:48:22.889 INFO:teuthology.orchestra.run.smithi076.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-12-10T18:48:22.901 INFO:teuthology.orchestra.run.smithi033.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-12-10T18:48:22.901 INFO:teuthology.orchestra.run.smithi033.stdout:uid [ unknown] Ceph.com (release key) 2023-12-10T18:48:22.902 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, rbd-mirror on remote deb x86_64 2023-12-10T18:48:22.905 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=3d2db44fa991bc355074aa9b7647f63b77201fef 2023-12-10T18:48:22.916 INFO:teuthology.orchestra.run.smithi076.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-12-10T18:48:22.916 INFO:teuthology.orchestra.run.smithi076.stdout:uid [ unknown] Ceph.com (release key) 2023-12-10T18:48:22.917 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, rbd-mirror on remote deb x86_64 2023-12-10T18:48:22.918 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=3d2db44fa991bc355074aa9b7647f63b77201fef 2023-12-10T18:48:23.050 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default/ 2023-12-10T18:48:23.050 INFO:teuthology.task.install.deb:Package version is 19.0.0-130-g3d2db44f-1jammy 2023-12-10T18:48:23.055 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default/ 2023-12-10T18:48:23.055 INFO:teuthology.task.install.deb:Package version is 19.0.0-130-g3d2db44f-1jammy 2023-12-10T18:48:23.170 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-10T18:48:23.170 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-12-10T18:48:23.173 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2023-12-10T18:48:23.173 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-12-10T18:48:23.186 DEBUG:teuthology.orchestra.run.smithi033:> sudo apt-get update 2023-12-10T18:48:23.187 DEBUG:teuthology.orchestra.run.smithi076:> sudo apt-get update 2023-12-10T18:48:23.357 INFO:teuthology.orchestra.run.smithi076.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-12-10T18:48:23.371 INFO:teuthology.orchestra.run.smithi076.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-12-10T18:48:23.390 INFO:teuthology.orchestra.run.smithi076.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-12-10T18:48:23.470 INFO:teuthology.orchestra.run.smithi076.stdout:Ign:4 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy InRelease 2023-12-10T18:48:23.475 INFO:teuthology.orchestra.run.smithi033.stdout:Ign:1 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy InRelease 2023-12-10T18:48:23.488 INFO:teuthology.orchestra.run.smithi033.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-12-10T18:48:23.488 INFO:teuthology.orchestra.run.smithi033.stdout:Hit:3 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-12-10T18:48:23.509 INFO:teuthology.orchestra.run.smithi076.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2023-12-10T18:48:23.513 INFO:teuthology.orchestra.run.smithi033.stdout:Get:4 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2023-12-10T18:48:23.546 INFO:teuthology.orchestra.run.smithi076.stdout:Ign:6 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy Release.gpg 2023-12-10T18:48:23.550 INFO:teuthology.orchestra.run.smithi033.stdout:Ign:5 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy Release.gpg 2023-12-10T18:48:23.561 INFO:teuthology.orchestra.run.smithi033.stdout:Hit:6 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-12-10T18:48:23.583 INFO:teuthology.orchestra.run.smithi076.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.7 kB] 2023-12-10T18:48:23.586 INFO:teuthology.orchestra.run.smithi033.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,399 B] 2023-12-10T18:48:23.625 INFO:teuthology.orchestra.run.smithi076.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,399 B] 2023-12-10T18:48:23.629 INFO:teuthology.orchestra.run.smithi033.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.7 kB] 2023-12-10T18:48:23.643 INFO:teuthology.orchestra.run.smithi033.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-12-10T18:48:23.864 INFO:teuthology.orchestra.run.smithi033.stdout:Fetched 28.8 kB in 0s (58.8 kB/s) 2023-12-10T18:48:23.899 INFO:teuthology.orchestra.run.smithi076.stdout:Hit:9 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-12-10T18:48:24.059 INFO:teuthology.orchestra.run.smithi076.stdout:Fetched 28.8 kB in 1s (42.1 kB/s) 2023-12-10T18:48:24.788 INFO:teuthology.orchestra.run.smithi033.stdout:Reading package lists... 2023-12-10T18:48:24.803 DEBUG:teuthology.orchestra.run.smithi033:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-130-g3d2db44f-1jammy cephadm=19.0.0-130-g3d2db44f-1jammy ceph-mds=19.0.0-130-g3d2db44f-1jammy ceph-mgr=19.0.0-130-g3d2db44f-1jammy ceph-common=19.0.0-130-g3d2db44f-1jammy ceph-fuse=19.0.0-130-g3d2db44f-1jammy ceph-test=19.0.0-130-g3d2db44f-1jammy ceph-volume=19.0.0-130-g3d2db44f-1jammy radosgw=19.0.0-130-g3d2db44f-1jammy python3-rados=19.0.0-130-g3d2db44f-1jammy python3-rgw=19.0.0-130-g3d2db44f-1jammy python3-cephfs=19.0.0-130-g3d2db44f-1jammy python3-rbd=19.0.0-130-g3d2db44f-1jammy libcephfs2=19.0.0-130-g3d2db44f-1jammy libcephfs-dev=19.0.0-130-g3d2db44f-1jammy librados2=19.0.0-130-g3d2db44f-1jammy librbd1=19.0.0-130-g3d2db44f-1jammy rbd-fuse=19.0.0-130-g3d2db44f-1jammy rbd-mirror=19.0.0-130-g3d2db44f-1jammy 2023-12-10T18:48:24.855 INFO:teuthology.orchestra.run.smithi033.stdout:Reading package lists... 2023-12-10T18:48:24.973 INFO:teuthology.orchestra.run.smithi076.stdout:Reading package lists... 2023-12-10T18:48:24.989 DEBUG:teuthology.orchestra.run.smithi076:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-130-g3d2db44f-1jammy cephadm=19.0.0-130-g3d2db44f-1jammy ceph-mds=19.0.0-130-g3d2db44f-1jammy ceph-mgr=19.0.0-130-g3d2db44f-1jammy ceph-common=19.0.0-130-g3d2db44f-1jammy ceph-fuse=19.0.0-130-g3d2db44f-1jammy ceph-test=19.0.0-130-g3d2db44f-1jammy ceph-volume=19.0.0-130-g3d2db44f-1jammy radosgw=19.0.0-130-g3d2db44f-1jammy python3-rados=19.0.0-130-g3d2db44f-1jammy python3-rgw=19.0.0-130-g3d2db44f-1jammy python3-cephfs=19.0.0-130-g3d2db44f-1jammy python3-rbd=19.0.0-130-g3d2db44f-1jammy libcephfs2=19.0.0-130-g3d2db44f-1jammy libcephfs-dev=19.0.0-130-g3d2db44f-1jammy librados2=19.0.0-130-g3d2db44f-1jammy librbd1=19.0.0-130-g3d2db44f-1jammy rbd-fuse=19.0.0-130-g3d2db44f-1jammy rbd-mirror=19.0.0-130-g3d2db44f-1jammy 2023-12-10T18:48:25.026 INFO:teuthology.orchestra.run.smithi033.stdout:Building dependency tree... 2023-12-10T18:48:25.027 INFO:teuthology.orchestra.run.smithi033.stdout:Reading state information... 2023-12-10T18:48:25.045 INFO:teuthology.orchestra.run.smithi076.stdout:Reading package lists... 2023-12-10T18:48:25.184 INFO:teuthology.orchestra.run.smithi033.stdout:The following packages were automatically installed and are no longer required: 2023-12-10T18:48:25.184 INFO:teuthology.orchestra.run.smithi033.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-12-10T18:48:25.184 INFO:teuthology.orchestra.run.smithi033.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-12-10T18:48:25.184 INFO:teuthology.orchestra.run.smithi033.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-12-10T18:48:25.184 INFO:teuthology.orchestra.run.smithi033.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-12-10T18:48:25.185 INFO:teuthology.orchestra.run.smithi033.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-12-10T18:48:25.185 INFO:teuthology.orchestra.run.smithi033.stdout:Use 'sudo apt autoremove' to remove them. 2023-12-10T18:48:25.185 INFO:teuthology.orchestra.run.smithi033.stdout:The following additional packages will be installed: 2023-12-10T18:48:25.185 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-12-10T18:48:25.185 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2023-12-10T18:48:25.185 INFO:teuthology.orchestra.run.smithi033.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2023-12-10T18:48:25.186 INFO:teuthology.orchestra.run.smithi033.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2023-12-10T18:48:25.187 INFO:teuthology.orchestra.run.smithi033.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2023-12-10T18:48:25.187 INFO:teuthology.orchestra.run.smithi033.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2023-12-10T18:48:25.187 INFO:teuthology.orchestra.run.smithi033.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2023-12-10T18:48:25.187 INFO:teuthology.orchestra.run.smithi033.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2023-12-10T18:48:25.187 INFO:teuthology.orchestra.run.smithi033.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2023-12-10T18:48:25.187 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2023-12-10T18:48:25.187 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2023-12-10T18:48:25.187 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2023-12-10T18:48:25.187 INFO:teuthology.orchestra.run.smithi033.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2023-12-10T18:48:25.187 INFO:teuthology.orchestra.run.smithi033.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2023-12-10T18:48:25.187 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2023-12-10T18:48:25.187 INFO:teuthology.orchestra.run.smithi033.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-12-10T18:48:25.187 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2023-12-10T18:48:25.188 INFO:teuthology.orchestra.run.smithi033.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2023-12-10T18:48:25.188 INFO:teuthology.orchestra.run.smithi033.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-12-10T18:48:25.188 INFO:teuthology.orchestra.run.smithi033.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-12-10T18:48:25.188 INFO:teuthology.orchestra.run.smithi033.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-12-10T18:48:25.188 INFO:teuthology.orchestra.run.smithi033.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-12-10T18:48:25.188 INFO:teuthology.orchestra.run.smithi033.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2023-12-10T18:48:25.188 INFO:teuthology.orchestra.run.smithi033.stdout: xmlstarlet zip 2023-12-10T18:48:25.188 INFO:teuthology.orchestra.run.smithi033.stdout:Suggested packages: 2023-12-10T18:48:25.188 INFO:teuthology.orchestra.run.smithi033.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2023-12-10T18:48:25.188 INFO:teuthology.orchestra.run.smithi033.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2023-12-10T18:48:25.188 INFO:teuthology.orchestra.run.smithi033.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2023-12-10T18:48:25.189 INFO:teuthology.orchestra.run.smithi033.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2023-12-10T18:48:25.189 INFO:teuthology.orchestra.run.smithi033.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2023-12-10T18:48:25.189 INFO:teuthology.orchestra.run.smithi033.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2023-12-10T18:48:25.189 INFO:teuthology.orchestra.run.smithi033.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2023-12-10T18:48:25.189 INFO:teuthology.orchestra.run.smithi033.stdout:Recommended packages: 2023-12-10T18:48:25.189 INFO:teuthology.orchestra.run.smithi033.stdout: btrfs-tools 2023-12-10T18:48:25.233 INFO:teuthology.orchestra.run.smithi076.stdout:Building dependency tree... 2023-12-10T18:48:25.234 INFO:teuthology.orchestra.run.smithi076.stdout:Reading state information... 2023-12-10T18:48:25.290 INFO:teuthology.orchestra.run.smithi033.stdout:The following NEW packages will be installed: 2023-12-10T18:48:25.290 INFO:teuthology.orchestra.run.smithi033.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-12-10T18:48:25.290 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-12-10T18:48:25.290 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2023-12-10T18:48:25.292 INFO:teuthology.orchestra.run.smithi033.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2023-12-10T18:48:25.292 INFO:teuthology.orchestra.run.smithi033.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2023-12-10T18:48:25.292 INFO:teuthology.orchestra.run.smithi033.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2023-12-10T18:48:25.292 INFO:teuthology.orchestra.run.smithi033.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2023-12-10T18:48:25.292 INFO:teuthology.orchestra.run.smithi033.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2023-12-10T18:48:25.292 INFO:teuthology.orchestra.run.smithi033.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2023-12-10T18:48:25.292 INFO:teuthology.orchestra.run.smithi033.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2023-12-10T18:48:25.292 INFO:teuthology.orchestra.run.smithi033.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2023-12-10T18:48:25.292 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2023-12-10T18:48:25.292 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2023-12-10T18:48:25.292 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2023-12-10T18:48:25.293 INFO:teuthology.orchestra.run.smithi033.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2023-12-10T18:48:25.293 INFO:teuthology.orchestra.run.smithi033.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2023-12-10T18:48:25.293 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2023-12-10T18:48:25.293 INFO:teuthology.orchestra.run.smithi033.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-12-10T18:48:25.293 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2023-12-10T18:48:25.293 INFO:teuthology.orchestra.run.smithi033.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2023-12-10T18:48:25.293 INFO:teuthology.orchestra.run.smithi033.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2023-12-10T18:48:25.293 INFO:teuthology.orchestra.run.smithi033.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2023-12-10T18:48:25.293 INFO:teuthology.orchestra.run.smithi033.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2023-12-10T18:48:25.293 INFO:teuthology.orchestra.run.smithi033.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2023-12-10T18:48:25.293 INFO:teuthology.orchestra.run.smithi033.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2023-12-10T18:48:25.293 INFO:teuthology.orchestra.run.smithi033.stdout: radosgw rbd-fuse rbd-mirror socat unzip xmlstarlet zip 2023-12-10T18:48:25.402 INFO:teuthology.orchestra.run.smithi076.stdout:The following packages were automatically installed and are no longer required: 2023-12-10T18:48:25.402 INFO:teuthology.orchestra.run.smithi076.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-12-10T18:48:25.402 INFO:teuthology.orchestra.run.smithi076.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-12-10T18:48:25.402 INFO:teuthology.orchestra.run.smithi076.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-12-10T18:48:25.402 INFO:teuthology.orchestra.run.smithi076.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-12-10T18:48:25.402 INFO:teuthology.orchestra.run.smithi076.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-12-10T18:48:25.403 INFO:teuthology.orchestra.run.smithi076.stdout:Use 'sudo apt autoremove' to remove them. 2023-12-10T18:48:25.403 INFO:teuthology.orchestra.run.smithi076.stdout:The following additional packages will be installed: 2023-12-10T18:48:25.403 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-12-10T18:48:25.404 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2023-12-10T18:48:25.404 INFO:teuthology.orchestra.run.smithi076.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2023-12-10T18:48:25.404 INFO:teuthology.orchestra.run.smithi076.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2023-12-10T18:48:25.404 INFO:teuthology.orchestra.run.smithi076.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2023-12-10T18:48:25.404 INFO:teuthology.orchestra.run.smithi076.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2023-12-10T18:48:25.404 INFO:teuthology.orchestra.run.smithi076.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2023-12-10T18:48:25.404 INFO:teuthology.orchestra.run.smithi076.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2023-12-10T18:48:25.404 INFO:teuthology.orchestra.run.smithi076.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2023-12-10T18:48:25.404 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2023-12-10T18:48:25.404 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2023-12-10T18:48:25.404 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2023-12-10T18:48:25.404 INFO:teuthology.orchestra.run.smithi076.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2023-12-10T18:48:25.405 INFO:teuthology.orchestra.run.smithi076.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2023-12-10T18:48:25.405 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2023-12-10T18:48:25.405 INFO:teuthology.orchestra.run.smithi076.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-12-10T18:48:25.405 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2023-12-10T18:48:25.405 INFO:teuthology.orchestra.run.smithi076.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2023-12-10T18:48:25.405 INFO:teuthology.orchestra.run.smithi076.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-12-10T18:48:25.405 INFO:teuthology.orchestra.run.smithi076.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-12-10T18:48:25.405 INFO:teuthology.orchestra.run.smithi076.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-12-10T18:48:25.405 INFO:teuthology.orchestra.run.smithi076.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-12-10T18:48:25.406 INFO:teuthology.orchestra.run.smithi076.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2023-12-10T18:48:25.406 INFO:teuthology.orchestra.run.smithi076.stdout: xmlstarlet zip 2023-12-10T18:48:25.406 INFO:teuthology.orchestra.run.smithi076.stdout:Suggested packages: 2023-12-10T18:48:25.406 INFO:teuthology.orchestra.run.smithi076.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2023-12-10T18:48:25.406 INFO:teuthology.orchestra.run.smithi076.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2023-12-10T18:48:25.406 INFO:teuthology.orchestra.run.smithi076.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2023-12-10T18:48:25.406 INFO:teuthology.orchestra.run.smithi076.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2023-12-10T18:48:25.406 INFO:teuthology.orchestra.run.smithi076.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2023-12-10T18:48:25.406 INFO:teuthology.orchestra.run.smithi076.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2023-12-10T18:48:25.406 INFO:teuthology.orchestra.run.smithi076.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2023-12-10T18:48:25.406 INFO:teuthology.orchestra.run.smithi076.stdout:Recommended packages: 2023-12-10T18:48:25.407 INFO:teuthology.orchestra.run.smithi076.stdout: btrfs-tools 2023-12-10T18:48:25.480 INFO:teuthology.orchestra.run.smithi033.stdout:0 upgraded, 119 newly installed, 0 to remove and 263 not upgraded. 2023-12-10T18:48:25.481 INFO:teuthology.orchestra.run.smithi033.stdout:Need to get 194 MB of archives. 2023-12-10T18:48:25.481 INFO:teuthology.orchestra.run.smithi033.stdout:After this operation, 873 MB of additional disk space will be used. 2023-12-10T18:48:25.481 INFO:teuthology.orchestra.run.smithi033.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2023-12-10T18:48:25.504 INFO:teuthology.orchestra.run.smithi076.stdout:The following NEW packages will be installed: 2023-12-10T18:48:25.504 INFO:teuthology.orchestra.run.smithi076.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-12-10T18:48:25.504 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-12-10T18:48:25.504 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2023-12-10T18:48:25.504 INFO:teuthology.orchestra.run.smithi076.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2023-12-10T18:48:25.504 INFO:teuthology.orchestra.run.smithi076.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2023-12-10T18:48:25.504 INFO:teuthology.orchestra.run.smithi076.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2023-12-10T18:48:25.505 INFO:teuthology.orchestra.run.smithi076.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2023-12-10T18:48:25.505 INFO:teuthology.orchestra.run.smithi076.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2023-12-10T18:48:25.505 INFO:teuthology.orchestra.run.smithi076.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2023-12-10T18:48:25.505 INFO:teuthology.orchestra.run.smithi076.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2023-12-10T18:48:25.505 INFO:teuthology.orchestra.run.smithi076.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2023-12-10T18:48:25.505 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2023-12-10T18:48:25.506 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2023-12-10T18:48:25.506 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2023-12-10T18:48:25.506 INFO:teuthology.orchestra.run.smithi076.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2023-12-10T18:48:25.506 INFO:teuthology.orchestra.run.smithi076.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2023-12-10T18:48:25.506 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2023-12-10T18:48:25.506 INFO:teuthology.orchestra.run.smithi076.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-12-10T18:48:25.506 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2023-12-10T18:48:25.506 INFO:teuthology.orchestra.run.smithi076.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2023-12-10T18:48:25.506 INFO:teuthology.orchestra.run.smithi076.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2023-12-10T18:48:25.506 INFO:teuthology.orchestra.run.smithi076.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2023-12-10T18:48:25.506 INFO:teuthology.orchestra.run.smithi076.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2023-12-10T18:48:25.506 INFO:teuthology.orchestra.run.smithi076.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2023-12-10T18:48:25.506 INFO:teuthology.orchestra.run.smithi076.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2023-12-10T18:48:25.506 INFO:teuthology.orchestra.run.smithi076.stdout: radosgw rbd-fuse rbd-mirror socat unzip xmlstarlet zip 2023-12-10T18:48:25.515 INFO:teuthology.orchestra.run.smithi033.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-130-g3d2db44f-1jammy [3,510 kB] 2023-12-10T18:48:25.693 INFO:teuthology.orchestra.run.smithi076.stdout:0 upgraded, 119 newly installed, 0 to remove and 263 not upgraded. 2023-12-10T18:48:25.693 INFO:teuthology.orchestra.run.smithi076.stdout:Need to get 194 MB of archives. 2023-12-10T18:48:25.693 INFO:teuthology.orchestra.run.smithi076.stdout:After this operation, 873 MB of additional disk space will be used. 2023-12-10T18:48:25.693 INFO:teuthology.orchestra.run.smithi076.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2023-12-10T18:48:25.725 INFO:teuthology.orchestra.run.smithi076.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-130-g3d2db44f-1jammy [3,510 kB] 2023-12-10T18:48:25.822 INFO:teuthology.orchestra.run.smithi033.stdout:Get:3 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-130-g3d2db44f-1jammy [3,215 kB] 2023-12-10T18:48:25.901 INFO:teuthology.orchestra.run.smithi033.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2023-12-10T18:48:25.913 INFO:teuthology.orchestra.run.smithi033.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2023-12-10T18:48:25.923 INFO:teuthology.orchestra.run.smithi033.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-130-g3d2db44f-1jammy [710 kB] 2023-12-10T18:48:25.945 INFO:teuthology.orchestra.run.smithi033.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-130-g3d2db44f-1jammy [356 kB] 2023-12-10T18:48:25.956 INFO:teuthology.orchestra.run.smithi033.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-130-g3d2db44f-1jammy [32.8 kB] 2023-12-10T18:48:25.958 INFO:teuthology.orchestra.run.smithi033.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-130-g3d2db44f-1jammy [180 kB] 2023-12-10T18:48:25.967 INFO:teuthology.orchestra.run.smithi033.stdout:Get:10 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-130-g3d2db44f-1jammy [63.5 kB] 2023-12-10T18:48:25.969 INFO:teuthology.orchestra.run.smithi033.stdout:Get:11 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-130-g3d2db44f-1jammy [330 kB] 2023-12-10T18:48:25.980 INFO:teuthology.orchestra.run.smithi033.stdout:Get:12 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-130-g3d2db44f-1jammy [6,226 kB] 2023-12-10T18:48:25.994 INFO:teuthology.orchestra.run.smithi033.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2023-12-10T18:48:26.121 INFO:teuthology.orchestra.run.smithi076.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2023-12-10T18:48:26.133 INFO:teuthology.orchestra.run.smithi076.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2023-12-10T18:48:26.193 INFO:teuthology.orchestra.run.smithi033.stdout:Get:14 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-130-g3d2db44f-1jammy [112 kB] 2023-12-10T18:48:26.195 INFO:teuthology.orchestra.run.smithi033.stdout:Get:15 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-130-g3d2db44f-1jammy [465 kB] 2023-12-10T18:48:26.210 INFO:teuthology.orchestra.run.smithi033.stdout:Get:16 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-130-g3d2db44f-1jammy [22.7 MB] 2023-12-10T18:48:26.214 INFO:teuthology.orchestra.run.smithi076.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2023-12-10T18:48:26.224 INFO:teuthology.orchestra.run.smithi033.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2023-12-10T18:48:26.236 INFO:teuthology.orchestra.run.smithi033.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2023-12-10T18:48:26.266 INFO:teuthology.orchestra.run.smithi033.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2023-12-10T18:48:26.275 INFO:teuthology.orchestra.run.smithi033.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2023-12-10T18:48:26.277 INFO:teuthology.orchestra.run.smithi033.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2023-12-10T18:48:26.277 INFO:teuthology.orchestra.run.smithi033.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2023-12-10T18:48:26.296 INFO:teuthology.orchestra.run.smithi033.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2023-12-10T18:48:26.444 INFO:teuthology.orchestra.run.smithi076.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2023-12-10T18:48:26.456 INFO:teuthology.orchestra.run.smithi076.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2023-12-10T18:48:26.465 INFO:teuthology.orchestra.run.smithi033.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2023-12-10T18:48:26.486 INFO:teuthology.orchestra.run.smithi076.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2023-12-10T18:48:26.495 INFO:teuthology.orchestra.run.smithi076.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2023-12-10T18:48:26.496 INFO:teuthology.orchestra.run.smithi076.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2023-12-10T18:48:26.497 INFO:teuthology.orchestra.run.smithi076.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2023-12-10T18:48:26.516 INFO:teuthology.orchestra.run.smithi076.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2023-12-10T18:48:26.684 INFO:teuthology.orchestra.run.smithi076.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2023-12-10T18:48:26.864 INFO:teuthology.orchestra.run.smithi033.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2023-12-10T18:48:26.898 INFO:teuthology.orchestra.run.smithi033.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2023-12-10T18:48:26.900 INFO:teuthology.orchestra.run.smithi033.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2023-12-10T18:48:26.979 INFO:teuthology.orchestra.run.smithi033.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.1 [174 kB] 2023-12-10T18:48:27.031 INFO:teuthology.orchestra.run.smithi033.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2023-12-10T18:48:27.092 INFO:teuthology.orchestra.run.smithi033.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2023-12-10T18:48:27.093 INFO:teuthology.orchestra.run.smithi033.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2023-12-10T18:48:27.093 INFO:teuthology.orchestra.run.smithi076.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2023-12-10T18:48:27.094 INFO:teuthology.orchestra.run.smithi033.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2023-12-10T18:48:27.094 INFO:teuthology.orchestra.run.smithi033.stdout:Get:33 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-130-g3d2db44f-1jammy [6,900 kB] 2023-12-10T18:48:27.126 INFO:teuthology.orchestra.run.smithi076.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2023-12-10T18:48:27.128 INFO:teuthology.orchestra.run.smithi076.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2023-12-10T18:48:27.150 INFO:teuthology.orchestra.run.smithi033.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2023-12-10T18:48:27.209 INFO:teuthology.orchestra.run.smithi076.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.1 [174 kB] 2023-12-10T18:48:27.261 INFO:teuthology.orchestra.run.smithi076.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2023-12-10T18:48:27.282 INFO:teuthology.orchestra.run.smithi033.stdout:Get:35 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-130-g3d2db44f-1jammy [238 kB] 2023-12-10T18:48:27.283 INFO:teuthology.orchestra.run.smithi076.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2023-12-10T18:48:27.288 INFO:teuthology.orchestra.run.smithi033.stdout:Get:36 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-130-g3d2db44f-1jammy [124 kB] 2023-12-10T18:48:27.289 INFO:teuthology.orchestra.run.smithi076.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2023-12-10T18:48:27.290 INFO:teuthology.orchestra.run.smithi076.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2023-12-10T18:48:27.293 INFO:teuthology.orchestra.run.smithi033.stdout:Get:37 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-130-g3d2db44f-1jammy [1,519 kB] 2023-12-10T18:48:27.302 INFO:teuthology.orchestra.run.smithi076.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2023-12-10T18:48:27.302 INFO:teuthology.orchestra.run.smithi076.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2023-12-10T18:48:27.315 INFO:teuthology.orchestra.run.smithi033.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2023-12-10T18:48:27.315 INFO:teuthology.orchestra.run.smithi033.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2023-12-10T18:48:27.350 INFO:teuthology.orchestra.run.smithi033.stdout:Get:40 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-130-g3d2db44f-1jammy [9,958 kB] 2023-12-10T18:48:27.407 INFO:teuthology.orchestra.run.smithi033.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2023-12-10T18:48:27.441 INFO:teuthology.orchestra.run.smithi033.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2023-12-10T18:48:27.468 INFO:teuthology.orchestra.run.smithi076.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2023-12-10T18:48:27.477 INFO:teuthology.orchestra.run.smithi033.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2023-12-10T18:48:27.493 INFO:teuthology.orchestra.run.smithi033.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2023-12-10T18:48:27.551 INFO:teuthology.orchestra.run.smithi076.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2023-12-10T18:48:27.567 INFO:teuthology.orchestra.run.smithi076.stdout:Get:26 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-130-g3d2db44f-1jammy [3,215 kB] 2023-12-10T18:48:27.580 INFO:teuthology.orchestra.run.smithi076.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2023-12-10T18:48:27.596 INFO:teuthology.orchestra.run.smithi076.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2023-12-10T18:48:27.641 INFO:teuthology.orchestra.run.smithi076.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2023-12-10T18:48:27.689 INFO:teuthology.orchestra.run.smithi033.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2023-12-10T18:48:27.704 INFO:teuthology.orchestra.run.smithi033.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2023-12-10T18:48:27.754 INFO:teuthology.orchestra.run.smithi033.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2023-12-10T18:48:27.798 INFO:teuthology.orchestra.run.smithi033.stdout:Get:48 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-130-g3d2db44f-1jammy [31.9 MB] 2023-12-10T18:48:27.798 INFO:teuthology.orchestra.run.smithi033.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2023-12-10T18:48:27.799 INFO:teuthology.orchestra.run.smithi033.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-12-10T18:48:27.851 INFO:teuthology.orchestra.run.smithi076.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2023-12-10T18:48:27.866 INFO:teuthology.orchestra.run.smithi076.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2023-12-10T18:48:27.874 INFO:teuthology.orchestra.run.smithi076.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2023-12-10T18:48:27.878 INFO:teuthology.orchestra.run.smithi076.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2023-12-10T18:48:27.900 INFO:teuthology.orchestra.run.smithi076.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-12-10T18:48:27.903 INFO:teuthology.orchestra.run.smithi076.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2023-12-10T18:48:27.920 INFO:teuthology.orchestra.run.smithi033.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2023-12-10T18:48:28.003 INFO:teuthology.orchestra.run.smithi033.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2023-12-10T18:48:28.069 INFO:teuthology.orchestra.run.smithi076.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2023-12-10T18:48:28.205 INFO:teuthology.orchestra.run.smithi033.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2023-12-10T18:48:28.261 INFO:teuthology.orchestra.run.smithi033.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2023-12-10T18:48:28.272 INFO:teuthology.orchestra.run.smithi033.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2023-12-10T18:48:28.325 INFO:teuthology.orchestra.run.smithi076.stdout:Get:37 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-130-g3d2db44f-1jammy [710 kB] 2023-12-10T18:48:28.402 INFO:teuthology.orchestra.run.smithi076.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2023-12-10T18:48:28.433 INFO:teuthology.orchestra.run.smithi076.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2023-12-10T18:48:28.437 INFO:teuthology.orchestra.run.smithi033.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2023-12-10T18:48:28.438 INFO:teuthology.orchestra.run.smithi033.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2023-12-10T18:48:28.443 INFO:teuthology.orchestra.run.smithi033.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2023-12-10T18:48:28.443 INFO:teuthology.orchestra.run.smithi076.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2023-12-10T18:48:28.448 INFO:teuthology.orchestra.run.smithi033.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2023-12-10T18:48:28.462 INFO:teuthology.orchestra.run.smithi033.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2023-12-10T18:48:28.484 INFO:teuthology.orchestra.run.smithi033.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2023-12-10T18:48:28.503 INFO:teuthology.orchestra.run.smithi076.stdout:Get:41 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-130-g3d2db44f-1jammy [356 kB] 2023-12-10T18:48:28.593 INFO:teuthology.orchestra.run.smithi076.stdout:Get:42 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-130-g3d2db44f-1jammy [32.8 kB] 2023-12-10T18:48:28.595 INFO:teuthology.orchestra.run.smithi076.stdout:Get:43 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-130-g3d2db44f-1jammy [180 kB] 2023-12-10T18:48:28.600 INFO:teuthology.orchestra.run.smithi076.stdout:Get:44 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-130-g3d2db44f-1jammy [63.5 kB] 2023-12-10T18:48:28.646 INFO:teuthology.orchestra.run.smithi076.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2023-12-10T18:48:28.647 INFO:teuthology.orchestra.run.smithi076.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2023-12-10T18:48:28.653 INFO:teuthology.orchestra.run.smithi076.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2023-12-10T18:48:28.654 INFO:teuthology.orchestra.run.smithi033.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2023-12-10T18:48:28.659 INFO:teuthology.orchestra.run.smithi076.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2023-12-10T18:48:28.674 INFO:teuthology.orchestra.run.smithi076.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2023-12-10T18:48:28.680 INFO:teuthology.orchestra.run.smithi076.stdout:Get:50 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-130-g3d2db44f-1jammy [330 kB] 2023-12-10T18:48:28.691 INFO:teuthology.orchestra.run.smithi076.stdout:Get:51 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-130-g3d2db44f-1jammy [6,226 kB] 2023-12-10T18:48:28.695 INFO:teuthology.orchestra.run.smithi076.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2023-12-10T18:48:28.702 INFO:teuthology.orchestra.run.smithi076.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2023-12-10T18:48:28.877 INFO:teuthology.orchestra.run.smithi076.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2023-12-10T18:48:28.994 INFO:teuthology.orchestra.run.smithi033.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2023-12-10T18:48:29.086 INFO:teuthology.orchestra.run.smithi033.stdout:Get:64 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-130-g3d2db44f-1jammy [14.0 kB] 2023-12-10T18:48:29.086 INFO:teuthology.orchestra.run.smithi033.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2023-12-10T18:48:29.140 INFO:teuthology.orchestra.run.smithi076.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2023-12-10T18:48:29.142 INFO:teuthology.orchestra.run.smithi033.stdout:Get:66 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-130-g3d2db44f-1jammy [865 kB] 2023-12-10T18:48:29.154 INFO:teuthology.orchestra.run.smithi033.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2023-12-10T18:48:29.159 INFO:teuthology.orchestra.run.smithi033.stdout:Get:68 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-130-g3d2db44f-1jammy [2,180 kB] 2023-12-10T18:48:29.170 INFO:teuthology.orchestra.run.smithi033.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2023-12-10T18:48:29.181 INFO:teuthology.orchestra.run.smithi033.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2023-12-10T18:48:29.200 INFO:teuthology.orchestra.run.smithi033.stdout:Get:71 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-130-g3d2db44f-1jammy [831 kB] 2023-12-10T18:48:29.212 INFO:teuthology.orchestra.run.smithi033.stdout:Get:72 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-130-g3d2db44f-1jammy [139 kB] 2023-12-10T18:48:29.214 INFO:teuthology.orchestra.run.smithi033.stdout:Get:73 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-130-g3d2db44f-1jammy [1,904 kB] 2023-12-10T18:48:29.228 INFO:teuthology.orchestra.run.smithi033.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2023-12-10T18:48:29.271 INFO:teuthology.orchestra.run.smithi033.stdout:Get:75 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-130-g3d2db44f-1jammy [8,625 kB] 2023-12-10T18:48:29.321 INFO:teuthology.orchestra.run.smithi076.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2023-12-10T18:48:29.333 INFO:teuthology.orchestra.run.smithi076.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2023-12-10T18:48:29.343 INFO:teuthology.orchestra.run.smithi076.stdout:Get:58 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-130-g3d2db44f-1jammy [112 kB] 2023-12-10T18:48:29.353 INFO:teuthology.orchestra.run.smithi076.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2023-12-10T18:48:29.378 INFO:teuthology.orchestra.run.smithi076.stdout:Get:60 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-130-g3d2db44f-1jammy [465 kB] 2023-12-10T18:48:29.418 INFO:teuthology.orchestra.run.smithi076.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2023-12-10T18:48:29.418 INFO:teuthology.orchestra.run.smithi076.stdout:Get:62 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-130-g3d2db44f-1jammy [22.7 MB] 2023-12-10T18:48:29.508 INFO:teuthology.orchestra.run.smithi033.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2023-12-10T18:48:29.510 INFO:teuthology.orchestra.run.smithi033.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jinja2 all 3.0.3-1 [108 kB] 2023-12-10T18:48:29.513 INFO:teuthology.orchestra.run.smithi033.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-12-10T18:48:29.513 INFO:teuthology.orchestra.run.smithi033.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2023-12-10T18:48:29.688 INFO:teuthology.orchestra.run.smithi033.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2023-12-10T18:48:29.694 INFO:teuthology.orchestra.run.smithi033.stdout:Get:81 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-130-g3d2db44f-1jammy [14.2 kB] 2023-12-10T18:48:29.695 INFO:teuthology.orchestra.run.smithi033.stdout:Get:82 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-130-g3d2db44f-1jammy [52.2 MB] 2023-12-10T18:48:29.705 INFO:teuthology.orchestra.run.smithi076.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2023-12-10T18:48:29.710 INFO:teuthology.orchestra.run.smithi076.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jinja2 all 3.0.3-1 [108 kB] 2023-12-10T18:48:29.712 INFO:teuthology.orchestra.run.smithi076.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-12-10T18:48:29.712 INFO:teuthology.orchestra.run.smithi076.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2023-12-10T18:48:29.714 INFO:teuthology.orchestra.run.smithi076.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2023-12-10T18:48:29.936 INFO:teuthology.orchestra.run.smithi076.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2023-12-10T18:48:30.373 INFO:teuthology.orchestra.run.smithi076.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2023-12-10T18:48:30.379 INFO:teuthology.orchestra.run.smithi076.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2023-12-10T18:48:30.391 INFO:teuthology.orchestra.run.smithi033.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2023-12-10T18:48:30.397 INFO:teuthology.orchestra.run.smithi033.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2023-12-10T18:48:30.398 INFO:teuthology.orchestra.run.smithi033.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2023-12-10T18:48:30.449 INFO:teuthology.orchestra.run.smithi033.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2023-12-10T18:48:30.473 INFO:teuthology.orchestra.run.smithi033.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2023-12-10T18:48:30.475 INFO:teuthology.orchestra.run.smithi033.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2023-12-10T18:48:30.476 INFO:teuthology.orchestra.run.smithi033.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2023-12-10T18:48:30.478 INFO:teuthology.orchestra.run.smithi033.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2023-12-10T18:48:30.480 INFO:teuthology.orchestra.run.smithi033.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2023-12-10T18:48:30.654 INFO:teuthology.orchestra.run.smithi033.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2023-12-10T18:48:30.670 INFO:teuthology.orchestra.run.smithi076.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2023-12-10T18:48:30.670 INFO:teuthology.orchestra.run.smithi076.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2023-12-10T18:48:30.673 INFO:teuthology.orchestra.run.smithi076.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2023-12-10T18:48:30.674 INFO:teuthology.orchestra.run.smithi076.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2023-12-10T18:48:30.676 INFO:teuthology.orchestra.run.smithi076.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2023-12-10T18:48:30.678 INFO:teuthology.orchestra.run.smithi076.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2023-12-10T18:48:30.679 INFO:teuthology.orchestra.run.smithi076.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2023-12-10T18:48:30.681 INFO:teuthology.orchestra.run.smithi076.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2023-12-10T18:48:30.786 INFO:teuthology.orchestra.run.smithi033.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2023-12-10T18:48:30.862 INFO:teuthology.orchestra.run.smithi076.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2023-12-10T18:48:31.083 INFO:teuthology.orchestra.run.smithi033.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2023-12-10T18:48:31.128 INFO:teuthology.orchestra.run.smithi033.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2023-12-10T18:48:31.154 INFO:teuthology.orchestra.run.smithi033.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2023-12-10T18:48:31.162 INFO:teuthology.orchestra.run.smithi033.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2023-12-10T18:48:31.207 INFO:teuthology.orchestra.run.smithi033.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2023-12-10T18:48:31.232 INFO:teuthology.orchestra.run.smithi033.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2023-12-10T18:48:31.239 INFO:teuthology.orchestra.run.smithi033.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2023-12-10T18:48:31.241 INFO:teuthology.orchestra.run.smithi033.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2023-12-10T18:48:31.266 INFO:teuthology.orchestra.run.smithi076.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2023-12-10T18:48:31.274 INFO:teuthology.orchestra.run.smithi033.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2023-12-10T18:48:31.313 INFO:teuthology.orchestra.run.smithi076.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2023-12-10T18:48:31.332 INFO:teuthology.orchestra.run.smithi076.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2023-12-10T18:48:31.440 INFO:teuthology.orchestra.run.smithi033.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2023-12-10T18:48:31.454 INFO:teuthology.orchestra.run.smithi076.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2023-12-10T18:48:31.456 INFO:teuthology.orchestra.run.smithi076.stdout:Get:84 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-130-g3d2db44f-1jammy [6,900 kB] 2023-12-10T18:48:31.484 INFO:teuthology.orchestra.run.smithi076.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2023-12-10T18:48:31.495 INFO:teuthology.orchestra.run.smithi076.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2023-12-10T18:48:31.500 INFO:teuthology.orchestra.run.smithi076.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2023-12-10T18:48:31.548 INFO:teuthology.orchestra.run.smithi076.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2023-12-10T18:48:31.551 INFO:teuthology.orchestra.run.smithi076.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2023-12-10T18:48:31.572 INFO:teuthology.orchestra.run.smithi076.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2023-12-10T18:48:31.737 INFO:teuthology.orchestra.run.smithi076.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-12-10T18:48:31.920 INFO:teuthology.orchestra.run.smithi033.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2023-12-10T18:48:31.921 INFO:teuthology.orchestra.run.smithi033.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-12-10T18:48:31.931 INFO:teuthology.orchestra.run.smithi033.stdout:Get:106 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2023-12-10T18:48:31.935 INFO:teuthology.orchestra.run.smithi033.stdout:Get:107 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2023-12-10T18:48:31.958 INFO:teuthology.orchestra.run.smithi076.stdout:Get:92 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-130-g3d2db44f-1jammy [238 kB] 2023-12-10T18:48:31.965 INFO:teuthology.orchestra.run.smithi033.stdout:Get:108 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2023-12-10T18:48:31.979 INFO:teuthology.orchestra.run.smithi033.stdout:Get:109 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2023-12-10T18:48:31.987 INFO:teuthology.orchestra.run.smithi076.stdout:Get:93 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-130-g3d2db44f-1jammy [124 kB] 2023-12-10T18:48:31.991 INFO:teuthology.orchestra.run.smithi076.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-130-g3d2db44f-1jammy [1,519 kB] 2023-12-10T18:48:32.002 INFO:teuthology.orchestra.run.smithi076.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2023-12-10T18:48:32.020 INFO:teuthology.orchestra.run.smithi076.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2023-12-10T18:48:32.105 INFO:teuthology.orchestra.run.smithi033.stdout:Get:110 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2023-12-10T18:48:32.107 INFO:teuthology.orchestra.run.smithi033.stdout:Get:111 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2023-12-10T18:48:32.108 INFO:teuthology.orchestra.run.smithi033.stdout:Get:112 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2023-12-10T18:48:32.111 INFO:teuthology.orchestra.run.smithi076.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-130-g3d2db44f-1jammy [9,958 kB] 2023-12-10T18:48:32.126 INFO:teuthology.orchestra.run.smithi033.stdout:Get:113 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2023-12-10T18:48:32.173 INFO:teuthology.orchestra.run.smithi076.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2023-12-10T18:48:32.202 INFO:teuthology.orchestra.run.smithi076.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2023-12-10T18:48:32.295 INFO:teuthology.orchestra.run.smithi033.stdout:Get:114 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2023-12-10T18:48:32.360 INFO:teuthology.orchestra.run.smithi076.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2023-12-10T18:48:32.363 INFO:teuthology.orchestra.run.smithi076.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2023-12-10T18:48:32.364 INFO:teuthology.orchestra.run.smithi076.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2023-12-10T18:48:32.386 INFO:teuthology.orchestra.run.smithi076.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2023-12-10T18:48:32.391 INFO:teuthology.orchestra.run.smithi076.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2023-12-10T18:48:32.727 INFO:teuthology.orchestra.run.smithi033.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-130-g3d2db44f-1jammy [120 kB] 2023-12-10T18:48:32.729 INFO:teuthology.orchestra.run.smithi033.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-130-g3d2db44f-1jammy [38.3 kB] 2023-12-10T18:48:32.730 INFO:teuthology.orchestra.run.smithi033.stdout:Get:117 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-130-g3d2db44f-1jammy [12.3 MB] 2023-12-10T18:48:32.889 INFO:teuthology.orchestra.run.smithi076.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-130-g3d2db44f-1jammy [31.9 MB] 2023-12-10T18:48:33.240 INFO:teuthology.orchestra.run.smithi033.stdout:Get:118 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-130-g3d2db44f-1jammy [91.7 kB] 2023-12-10T18:48:33.242 INFO:teuthology.orchestra.run.smithi033.stdout:Get:119 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 rbd-mirror amd64 19.0.0-130-g3d2db44f-1jammy [3,124 kB] 2023-12-10T18:48:34.003 INFO:teuthology.orchestra.run.smithi033.stdout:Fetched 194 MB in 8s (23.8 MB/s) 2023-12-10T18:48:34.156 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2023-12-10T18:48:34.258 INFO:teuthology.orchestra.run.smithi033.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165989 files and directories currently installed.) 2023-12-10T18:48:34.264 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2023-12-10T18:48:34.346 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-12-10T18:48:34.401 INFO:teuthology.orchestra.run.smithi076.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-130-g3d2db44f-1jammy [14.0 kB] 2023-12-10T18:48:34.401 INFO:teuthology.orchestra.run.smithi076.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-130-g3d2db44f-1jammy [865 kB] 2023-12-10T18:48:34.418 INFO:teuthology.orchestra.run.smithi076.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-130-g3d2db44f-1jammy [2,180 kB] 2023-12-10T18:48:34.457 INFO:teuthology.orchestra.run.smithi076.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-130-g3d2db44f-1jammy [831 kB] 2023-12-10T18:48:34.468 INFO:teuthology.orchestra.run.smithi076.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-130-g3d2db44f-1jammy [139 kB] 2023-12-10T18:48:34.470 INFO:teuthology.orchestra.run.smithi076.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-130-g3d2db44f-1jammy [1,904 kB] 2023-12-10T18:48:34.517 INFO:teuthology.orchestra.run.smithi076.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-130-g3d2db44f-1jammy [8,625 kB] 2023-12-10T18:48:34.692 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-12-10T18:48:34.715 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2023-12-10T18:48:34.750 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2023-12-10T18:48:34.801 INFO:teuthology.orchestra.run.smithi076.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-130-g3d2db44f-1jammy [14.2 kB] 2023-12-10T18:48:34.802 INFO:teuthology.orchestra.run.smithi076.stdout:Get:114 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-130-g3d2db44f-1jammy [52.2 MB] 2023-12-10T18:48:35.078 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-12-10T18:48:35.105 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2023-12-10T18:48:35.145 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-12-10T18:48:35.593 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-12-10T18:48:35.617 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-12-10T18:48:35.665 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-10T18:48:36.002 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-12-10T18:48:36.029 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-12-10T18:48:36.068 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-10T18:48:36.405 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-12-10T18:48:36.432 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-12-10T18:48:36.504 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-10T18:48:36.791 INFO:teuthology.orchestra.run.smithi076.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-130-g3d2db44f-1jammy [120 kB] 2023-12-10T18:48:36.794 INFO:teuthology.orchestra.run.smithi076.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-130-g3d2db44f-1jammy [38.3 kB] 2023-12-10T18:48:36.795 INFO:teuthology.orchestra.run.smithi076.stdout:Get:117 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-130-g3d2db44f-1jammy [12.3 MB] 2023-12-10T18:48:36.833 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2023-12-10T18:48:36.860 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2023-12-10T18:48:36.899 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-12-10T18:48:37.138 INFO:teuthology.orchestra.run.smithi076.stdout:Get:118 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-130-g3d2db44f-1jammy [91.7 kB] 2023-12-10T18:48:37.140 INFO:teuthology.orchestra.run.smithi076.stdout:Get:119 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 rbd-mirror amd64 19.0.0-130-g3d2db44f-1jammy [3,124 kB] 2023-12-10T18:48:37.161 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package librados2. 2023-12-10T18:48:37.177 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../007-librados2_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-10T18:48:37.211 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking librados2 (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:48:37.647 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package librbd1. 2023-12-10T18:48:37.674 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../008-librbd1_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-10T18:48:37.713 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking librbd1 (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:48:37.795 INFO:teuthology.orchestra.run.smithi076.stdout:Fetched 194 MB in 12s (16.5 MB/s) 2023-12-10T18:48:37.972 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2023-12-10T18:48:38.072 INFO:teuthology.orchestra.run.smithi076.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165989 files and directories currently installed.) 2023-12-10T18:48:38.077 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2023-12-10T18:48:38.162 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-12-10T18:48:38.192 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libcephfs2. 2023-12-10T18:48:38.219 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-10T18:48:38.425 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libcephfs2 (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:48:38.458 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-12-10T18:48:38.485 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2023-12-10T18:48:38.517 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2023-12-10T18:48:38.769 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-rados. 2023-12-10T18:48:38.785 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../010-python3-rados_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-10T18:48:38.827 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-rados (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:48:39.139 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-12-10T18:48:39.166 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-130-g3d2db44f-1jammy_all.deb ... 2023-12-10T18:48:39.205 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-ceph-argparse (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:48:39.287 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-12-10T18:48:39.314 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2023-12-10T18:48:39.362 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-12-10T18:48:39.466 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-cephfs. 2023-12-10T18:48:39.493 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-10T18:48:39.533 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-cephfs (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:48:39.808 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-12-10T18:48:39.819 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-ceph-common. 2023-12-10T18:48:39.834 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-12-10T18:48:39.835 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-130-g3d2db44f-1jammy_all.deb ... 2023-12-10T18:48:39.869 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-ceph-common (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:48:39.891 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-10T18:48:40.130 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-wcwidth. 2023-12-10T18:48:40.147 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2023-12-10T18:48:40.180 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2023-12-10T18:48:40.294 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-12-10T18:48:40.321 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-12-10T18:48:40.361 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-10T18:48:40.492 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-prettytable. 2023-12-10T18:48:40.508 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2023-12-10T18:48:40.550 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2023-12-10T18:48:40.715 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-12-10T18:48:40.741 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-12-10T18:48:40.781 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-10T18:48:40.836 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-rbd. 2023-12-10T18:48:40.863 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-10T18:48:40.903 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-rbd (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:48:41.077 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2023-12-10T18:48:41.103 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2023-12-10T18:48:41.135 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-12-10T18:48:41.214 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-12-10T18:48:41.241 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2023-12-10T18:48:41.280 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2023-12-10T18:48:41.505 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package librados2. 2023-12-10T18:48:41.525 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libreadline-dev:amd64. 2023-12-10T18:48:41.531 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../007-librados2_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-10T18:48:41.552 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2023-12-10T18:48:41.580 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking librados2 (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:48:41.591 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2023-12-10T18:48:41.894 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2023-12-10T18:48:41.911 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2023-12-10T18:48:41.953 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2023-12-10T18:48:42.058 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package librbd1. 2023-12-10T18:48:42.084 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../008-librbd1_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-10T18:48:42.125 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking librbd1 (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:48:42.281 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package lua5.1. 2023-12-10T18:48:42.308 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2023-12-10T18:48:42.349 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2023-12-10T18:48:42.586 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package libcephfs2. 2023-12-10T18:48:42.613 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-10T18:48:42.618 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package lua-any. 2023-12-10T18:48:42.634 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2023-12-10T18:48:42.653 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking libcephfs2 (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:48:42.676 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking lua-any (27ubuntu1) ... 2023-12-10T18:48:42.920 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package zip. 2023-12-10T18:48:42.936 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2023-12-10T18:48:42.973 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-rados. 2023-12-10T18:48:42.978 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking zip (3.0-12build2) ... 2023-12-10T18:48:42.999 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../010-python3-rados_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-10T18:48:43.047 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-rados (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:48:43.299 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package unzip. 2023-12-10T18:48:43.326 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.1_amd64.deb ... 2023-12-10T18:48:43.365 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking unzip (6.0-26ubuntu3.1) ... 2023-12-10T18:48:43.403 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-12-10T18:48:43.429 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-130-g3d2db44f-1jammy_all.deb ... 2023-12-10T18:48:43.469 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-ceph-argparse (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:48:43.720 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package luarocks. 2023-12-10T18:48:43.739 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-cephfs. 2023-12-10T18:48:43.747 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2023-12-10T18:48:43.754 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-10T18:48:43.778 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2023-12-10T18:48:43.805 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-cephfs (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:48:44.144 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package librgw2. 2023-12-10T18:48:44.172 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../025-librgw2_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-10T18:48:44.209 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-ceph-common. 2023-12-10T18:48:44.210 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking librgw2 (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:48:44.235 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-130-g3d2db44f-1jammy_all.deb ... 2023-12-10T18:48:44.275 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-ceph-common (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:48:44.738 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-rgw. 2023-12-10T18:48:44.746 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-wcwidth. 2023-12-10T18:48:44.754 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-10T18:48:44.772 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2023-12-10T18:48:44.798 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-rgw (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:48:44.812 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2023-12-10T18:48:45.091 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package liboath0:amd64. 2023-12-10T18:48:45.107 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2023-12-10T18:48:45.132 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-prettytable. 2023-12-10T18:48:45.141 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2023-12-10T18:48:45.158 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2023-12-10T18:48:45.199 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2023-12-10T18:48:45.444 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libradosstriper1. 2023-12-10T18:48:45.460 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-rbd. 2023-12-10T18:48:45.471 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-10T18:48:45.475 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-10T18:48:45.510 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-rbd (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:48:45.511 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libradosstriper1 (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:48:45.839 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-12-10T18:48:45.865 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2023-12-10T18:48:45.905 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2023-12-10T18:48:45.931 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph-common. 2023-12-10T18:48:45.958 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../029-ceph-common_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-10T18:48:45.997 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph-common (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:48:46.192 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package libreadline-dev:amd64. 2023-12-10T18:48:46.218 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2023-12-10T18:48:46.258 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2023-12-10T18:48:46.545 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2023-12-10T18:48:46.560 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2023-12-10T18:48:46.595 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2023-12-10T18:48:46.940 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package lua5.1. 2023-12-10T18:48:46.967 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2023-12-10T18:48:47.015 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2023-12-10T18:48:47.253 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph-base. 2023-12-10T18:48:47.280 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../030-ceph-base_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-10T18:48:47.329 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph-base (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:48:47.394 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package lua-any. 2023-12-10T18:48:47.420 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2023-12-10T18:48:47.461 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking lua-any (27ubuntu1) ... 2023-12-10T18:48:47.696 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package zip. 2023-12-10T18:48:47.711 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2023-12-10T18:48:47.755 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking zip (3.0-12build2) ... 2023-12-10T18:48:48.057 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-jaraco.functools. 2023-12-10T18:48:48.085 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2023-12-10T18:48:48.108 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package unzip. 2023-12-10T18:48:48.115 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2023-12-10T18:48:48.135 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.1_amd64.deb ... 2023-12-10T18:48:48.183 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking unzip (6.0-26ubuntu3.1) ... 2023-12-10T18:48:48.418 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-cheroot. 2023-12-10T18:48:48.445 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2023-12-10T18:48:48.485 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-12-10T18:48:48.630 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package luarocks. 2023-12-10T18:48:48.656 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2023-12-10T18:48:48.697 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2023-12-10T18:48:48.988 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-jaraco.classes. 2023-12-10T18:48:49.015 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2023-12-10T18:48:49.054 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2023-12-10T18:48:49.114 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package librgw2. 2023-12-10T18:48:49.140 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../025-librgw2_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-10T18:48:49.225 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking librgw2 (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:48:49.324 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-jaraco.text. 2023-12-10T18:48:49.341 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2023-12-10T18:48:49.374 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2023-12-10T18:48:49.677 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-jaraco.collections. 2023-12-10T18:48:49.705 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2023-12-10T18:48:49.742 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-rgw. 2023-12-10T18:48:49.744 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2023-12-10T18:48:49.768 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-10T18:48:49.808 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-rgw (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:48:50.047 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-tempora. 2023-12-10T18:48:50.076 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2023-12-10T18:48:50.113 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-tempora (4.1.2-1) ... 2023-12-10T18:48:50.137 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package liboath0:amd64. 2023-12-10T18:48:50.164 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2023-12-10T18:48:50.203 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2023-12-10T18:48:50.425 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-portend. 2023-12-10T18:48:50.442 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2023-12-10T18:48:50.473 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package libradosstriper1. 2023-12-10T18:48:50.483 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-portend (3.0.0-1) ... 2023-12-10T18:48:50.499 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-10T18:48:50.540 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking libradosstriper1 (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:48:50.728 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-zc.lockfile. 2023-12-10T18:48:50.754 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2023-12-10T18:48:50.786 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2023-12-10T18:48:50.943 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package ceph-common. 2023-12-10T18:48:50.970 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../029-ceph-common_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-10T18:48:51.010 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking ceph-common (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:48:51.089 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-cherrypy3. 2023-12-10T18:48:51.116 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2023-12-10T18:48:51.155 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2023-12-10T18:48:51.459 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-natsort. 2023-12-10T18:48:51.475 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2023-12-10T18:48:51.509 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-natsort (8.0.2-1) ... 2023-12-10T18:48:51.779 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-logutils. 2023-12-10T18:48:51.806 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2023-12-10T18:48:51.845 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-logutils (0.3.3-8) ... 2023-12-10T18:48:52.148 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-markupsafe. 2023-12-10T18:48:52.175 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2023-12-10T18:48:52.215 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2023-12-10T18:48:52.224 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package ceph-base. 2023-12-10T18:48:52.251 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../030-ceph-base_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-10T18:48:52.300 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking ceph-base (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:48:52.518 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-mako. 2023-12-10T18:48:52.545 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2023-12-10T18:48:52.585 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-12-10T18:48:52.888 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-simplegeneric. 2023-12-10T18:48:52.904 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2023-12-10T18:48:52.938 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-12-10T18:48:53.004 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-jaraco.functools. 2023-12-10T18:48:53.020 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2023-12-10T18:48:53.054 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2023-12-10T18:48:53.232 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-singledispatch. 2023-12-10T18:48:53.260 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2023-12-10T18:48:53.299 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2023-12-10T18:48:53.349 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-cheroot. 2023-12-10T18:48:53.378 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2023-12-10T18:48:53.416 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-12-10T18:48:53.585 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-webob. 2023-12-10T18:48:53.612 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2023-12-10T18:48:53.652 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2023-12-10T18:48:53.753 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-jaraco.classes. 2023-12-10T18:48:53.779 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2023-12-10T18:48:53.828 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2023-12-10T18:48:53.996 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-waitress. 2023-12-10T18:48:54.023 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2023-12-10T18:48:54.066 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2023-12-10T18:48:54.114 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-jaraco.text. 2023-12-10T18:48:54.140 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2023-12-10T18:48:54.172 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2023-12-10T18:48:54.400 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-tempita. 2023-12-10T18:48:54.427 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2023-12-10T18:48:54.467 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2023-12-10T18:48:54.642 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-jaraco.collections. 2023-12-10T18:48:54.669 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2023-12-10T18:48:54.709 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2023-12-10T18:48:54.769 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-paste. 2023-12-10T18:48:54.796 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2023-12-10T18:48:54.836 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2023-12-10T18:48:55.021 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-tempora. 2023-12-10T18:48:55.047 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2023-12-10T18:48:55.087 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-tempora (4.1.2-1) ... 2023-12-10T18:48:55.130 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-12-10T18:48:55.147 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2023-12-10T18:48:55.189 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2023-12-10T18:48:55.374 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-portend. 2023-12-10T18:48:55.400 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2023-12-10T18:48:55.432 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-portend (3.0.0-1) ... 2023-12-10T18:48:55.494 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-pastedeploy. 2023-12-10T18:48:55.521 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2023-12-10T18:48:55.560 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2023-12-10T18:48:55.735 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-zc.lockfile. 2023-12-10T18:48:55.762 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2023-12-10T18:48:55.802 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2023-12-10T18:48:55.822 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-webtest. 2023-12-10T18:48:55.849 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2023-12-10T18:48:55.880 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-webtest (2.0.35-1) ... 2023-12-10T18:48:56.131 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-cherrypy3. 2023-12-10T18:48:56.157 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2023-12-10T18:48:56.182 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-pecan. 2023-12-10T18:48:56.197 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2023-12-10T18:48:56.210 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2023-12-10T18:48:56.249 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2023-12-10T18:48:56.535 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-natsort. 2023-12-10T18:48:56.561 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2023-12-10T18:48:56.601 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-natsort (8.0.2-1) ... 2023-12-10T18:48:56.629 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-certifi. 2023-12-10T18:48:56.656 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2023-12-10T18:48:56.704 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2023-12-10T18:48:56.921 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-logutils. 2023-12-10T18:48:56.948 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2023-12-10T18:48:56.988 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-logutils (0.3.3-8) ... 2023-12-10T18:48:57.023 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-idna. 2023-12-10T18:48:57.051 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2023-12-10T18:48:57.090 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-idna (3.3-1) ... 2023-12-10T18:48:57.291 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-markupsafe. 2023-12-10T18:48:57.318 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2023-12-10T18:48:57.358 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2023-12-10T18:48:57.392 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-urllib3. 2023-12-10T18:48:57.420 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2023-12-10T18:48:57.459 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2023-12-10T18:48:57.678 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-mako. 2023-12-10T18:48:57.704 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2023-12-10T18:48:57.744 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-12-10T18:48:57.787 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-requests. 2023-12-10T18:48:57.815 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2023-12-10T18:48:57.854 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-12-10T18:48:58.056 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-simplegeneric. 2023-12-10T18:48:58.083 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2023-12-10T18:48:58.123 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-12-10T18:48:58.140 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-werkzeug. 2023-12-10T18:48:58.168 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2023-12-10T18:48:58.227 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-12-10T18:48:58.434 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-singledispatch. 2023-12-10T18:48:58.461 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2023-12-10T18:48:58.501 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2023-12-10T18:48:58.518 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-12-10T18:48:58.535 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-130-g3d2db44f-1jammy_all.deb ... 2023-12-10T18:48:58.576 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph-mgr-modules-core (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:48:59.542 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-webob. 2023-12-10T18:48:59.565 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2023-12-10T18:48:59.605 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2023-12-10T18:48:59.823 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-12-10T18:48:59.850 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-10T18:48:59.881 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:48:59.925 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-waitress. 2023-12-10T18:48:59.954 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2023-12-10T18:48:59.995 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2023-12-10T18:49:00.193 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-bcrypt. 2023-12-10T18:49:00.220 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2023-12-10T18:49:00.259 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2023-12-10T18:49:00.303 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-tempita. 2023-12-10T18:49:00.330 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2023-12-10T18:49:00.378 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2023-12-10T18:49:00.562 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-openssl. 2023-12-10T18:49:00.589 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2023-12-10T18:49:00.628 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-openssl (21.0.0-1) ... 2023-12-10T18:49:00.681 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-paste. 2023-12-10T18:49:00.709 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2023-12-10T18:49:00.748 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2023-12-10T18:49:00.931 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph-mgr. 2023-12-10T18:49:00.959 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-10T18:49:00.998 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph-mgr (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:49:01.077 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-12-10T18:49:01.103 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2023-12-10T18:49:01.143 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2023-12-10T18:49:01.359 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph-mon. 2023-12-10T18:49:01.390 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-10T18:49:01.426 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph-mon (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:49:01.430 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-pastedeploy. 2023-12-10T18:49:01.456 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2023-12-10T18:49:01.496 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2023-12-10T18:49:01.758 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-webtest. 2023-12-10T18:49:01.784 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2023-12-10T18:49:01.816 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-webtest (2.0.35-1) ... 2023-12-10T18:49:02.120 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-pecan. 2023-12-10T18:49:02.146 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2023-12-10T18:49:02.186 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2023-12-10T18:49:02.380 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph-osd. 2023-12-10T18:49:02.407 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-10T18:49:02.446 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph-osd (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:49:02.558 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-certifi. 2023-12-10T18:49:02.585 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2023-12-10T18:49:02.625 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2023-12-10T18:49:02.945 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-idna. 2023-12-10T18:49:02.971 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2023-12-10T18:49:03.011 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-idna (3.3-1) ... 2023-12-10T18:49:03.290 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-urllib3. 2023-12-10T18:49:03.317 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2023-12-10T18:49:03.348 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2023-12-10T18:49:03.638 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-requests. 2023-12-10T18:49:03.650 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2023-12-10T18:49:03.685 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-12-10T18:49:03.964 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-werkzeug. 2023-12-10T18:49:03.990 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2023-12-10T18:49:04.030 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-12-10T18:49:04.300 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-12-10T18:49:04.316 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-130-g3d2db44f-1jammy_all.deb ... 2023-12-10T18:49:04.332 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph. 2023-12-10T18:49:04.348 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../066-ceph_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-10T18:49:04.350 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking ceph-mgr-modules-core (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:49:04.390 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:49:04.706 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-12-10T18:49:04.733 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-10T18:49:04.743 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph-fuse. 2023-12-10T18:49:04.770 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-10T18:49:04.772 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:49:04.810 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph-fuse (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:49:05.146 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph-mds. 2023-12-10T18:49:05.173 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-10T18:49:05.204 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph-mds (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:49:05.268 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-bcrypt. 2023-12-10T18:49:05.294 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2023-12-10T18:49:05.334 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2023-12-10T18:49:05.579 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-openssl. 2023-12-10T18:49:05.599 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package cephadm. 2023-12-10T18:49:05.605 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2023-12-10T18:49:05.626 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../069-cephadm_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-10T18:49:05.645 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-openssl (21.0.0-1) ... 2023-12-10T18:49:05.665 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking cephadm (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:49:05.984 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-asyncssh. 2023-12-10T18:49:06.011 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2023-12-10T18:49:06.024 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package ceph-mgr. 2023-12-10T18:49:06.050 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-10T18:49:06.059 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2023-12-10T18:49:06.090 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking ceph-mgr (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:49:06.370 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python-babel-localedata. 2023-12-10T18:49:06.386 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2023-12-10T18:49:06.429 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-12-10T18:49:06.486 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package ceph-mon. 2023-12-10T18:49:06.512 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-10T18:49:06.552 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking ceph-mon (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:49:07.448 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package ceph-osd. 2023-12-10T18:49:07.467 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-babel. 2023-12-10T18:49:07.474 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-10T18:49:07.494 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2023-12-10T18:49:07.514 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking ceph-osd (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:49:07.533 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2023-12-10T18:49:07.828 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-jinja2. 2023-12-10T18:49:07.855 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1_all.deb ... 2023-12-10T18:49:07.894 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-jinja2 (3.0.3-1) ... 2023-12-10T18:49:08.214 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-12-10T18:49:08.242 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-130-g3d2db44f-1jammy_all.deb ... 2023-12-10T18:49:08.281 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph-mgr-cephadm (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:49:08.651 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-repoze.lru. 2023-12-10T18:49:08.678 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2023-12-10T18:49:08.717 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-12-10T18:49:08.978 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-routes. 2023-12-10T18:49:08.995 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2023-12-10T18:49:09.028 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2023-12-10T18:49:09.331 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-12-10T18:49:09.358 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-130-g3d2db44f-1jammy_all.deb ... 2023-12-10T18:49:09.397 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph-mgr-dashboard (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:49:09.426 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package ceph. 2023-12-10T18:49:09.442 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../066-ceph_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-10T18:49:09.485 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking ceph (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:49:09.813 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package ceph-fuse. 2023-12-10T18:49:09.840 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-10T18:49:09.880 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking ceph-fuse (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:49:09.926 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2023-12-10T18:49:09.953 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2023-12-10T18:49:09.992 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-12-10T18:49:10.267 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package ceph-mds. 2023-12-10T18:49:10.293 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-10T18:49:10.333 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking ceph-mds (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:49:10.496 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-joblib. 2023-12-10T18:49:10.523 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2023-12-10T18:49:10.562 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2023-12-10T18:49:10.703 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package cephadm. 2023-12-10T18:49:10.729 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../069-cephadm_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-10T18:49:10.761 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking cephadm (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:49:10.932 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-threadpoolctl. 2023-12-10T18:49:10.960 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2023-12-10T18:49:10.999 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2023-12-10T18:49:11.073 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-asyncssh. 2023-12-10T18:49:11.099 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2023-12-10T18:49:11.139 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2023-12-10T18:49:11.335 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-sklearn. 2023-12-10T18:49:11.362 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2023-12-10T18:49:11.401 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2023-12-10T18:49:11.459 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python-babel-localedata. 2023-12-10T18:49:11.475 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2023-12-10T18:49:11.509 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-12-10T18:49:12.085 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-12-10T18:49:12.113 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-130-g3d2db44f-1jammy_all.deb ... 2023-12-10T18:49:12.237 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:49:12.598 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-babel. 2023-12-10T18:49:12.626 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2023-12-10T18:49:12.664 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2023-12-10T18:49:12.976 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-jinja2. 2023-12-10T18:49:13.004 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1_all.deb ... 2023-12-10T18:49:13.043 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-jinja2 (3.0.3-1) ... 2023-12-10T18:49:13.198 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-cachetools. 2023-12-10T18:49:13.226 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2023-12-10T18:49:13.265 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2023-12-10T18:49:13.388 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-12-10T18:49:13.404 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-130-g3d2db44f-1jammy_all.deb ... 2023-12-10T18:49:13.438 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking ceph-mgr-cephadm (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:49:13.567 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-pyasn1. 2023-12-10T18:49:13.595 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2023-12-10T18:49:13.634 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2023-12-10T18:49:13.758 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-repoze.lru. 2023-12-10T18:49:13.774 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2023-12-10T18:49:13.808 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-12-10T18:49:13.971 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-12-10T18:49:13.998 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2023-12-10T18:49:14.037 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2023-12-10T18:49:14.078 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-routes. 2023-12-10T18:49:14.105 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2023-12-10T18:49:14.144 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2023-12-10T18:49:14.357 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-rsa. 2023-12-10T18:49:14.385 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2023-12-10T18:49:14.423 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-rsa (4.8-1) ... 2023-12-10T18:49:14.424 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-12-10T18:49:14.450 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-130-g3d2db44f-1jammy_all.deb ... 2023-12-10T18:49:14.490 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking ceph-mgr-dashboard (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:49:14.711 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-google-auth. 2023-12-10T18:49:14.738 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2023-12-10T18:49:14.769 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2023-12-10T18:49:14.943 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2023-12-10T18:49:14.960 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2023-12-10T18:49:15.002 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-12-10T18:49:15.038 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-12-10T18:49:15.056 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2023-12-10T18:49:15.088 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-12-10T18:49:15.392 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-websocket. 2023-12-10T18:49:15.419 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2023-12-10T18:49:15.458 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-websocket (1.2.3-1) ... 2023-12-10T18:49:15.664 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-joblib. 2023-12-10T18:49:15.691 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2023-12-10T18:49:15.731 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2023-12-10T18:49:15.795 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-kubernetes. 2023-12-10T18:49:15.822 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2023-12-10T18:49:15.861 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2023-12-10T18:49:16.085 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-threadpoolctl. 2023-12-10T18:49:16.111 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2023-12-10T18:49:16.143 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2023-12-10T18:49:16.438 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-sklearn. 2023-12-10T18:49:16.465 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2023-12-10T18:49:16.504 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2023-12-10T18:49:16.525 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-12-10T18:49:16.541 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-130-g3d2db44f-1jammy_all.deb ... 2023-12-10T18:49:16.583 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:49:16.902 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libonig5:amd64. 2023-12-10T18:49:16.930 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2023-12-10T18:49:16.969 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2023-12-10T18:49:17.198 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-12-10T18:49:17.225 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-130-g3d2db44f-1jammy_all.deb ... 2023-12-10T18:49:17.247 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libjq1:amd64. 2023-12-10T18:49:17.275 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2023-12-10T18:49:17.305 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-12-10T18:49:17.350 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:49:17.558 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package jq. 2023-12-10T18:49:17.585 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2023-12-10T18:49:17.624 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2023-12-10T18:49:17.877 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package socat. 2023-12-10T18:49:17.893 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2023-12-10T18:49:17.943 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2023-12-10T18:49:18.262 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-cachetools. 2023-12-10T18:49:18.263 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package xmlstarlet. 2023-12-10T18:49:18.278 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2023-12-10T18:49:18.280 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2023-12-10T18:49:18.312 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2023-12-10T18:49:18.321 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2023-12-10T18:49:18.598 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-pyasn1. 2023-12-10T18:49:18.626 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2023-12-10T18:49:18.665 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2023-12-10T18:49:18.692 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph-test. 2023-12-10T18:49:18.717 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../097-ceph-test_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-10T18:49:18.758 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph-test (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:49:18.994 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-12-10T18:49:19.021 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2023-12-10T18:49:19.060 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2023-12-10T18:49:19.397 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-rsa. 2023-12-10T18:49:19.424 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2023-12-10T18:49:19.472 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-rsa (4.8-1) ... 2023-12-10T18:49:19.743 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-google-auth. 2023-12-10T18:49:19.770 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2023-12-10T18:49:19.801 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2023-12-10T18:49:20.121 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-12-10T18:49:20.149 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2023-12-10T18:49:20.188 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-12-10T18:49:20.450 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-websocket. 2023-12-10T18:49:20.477 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2023-12-10T18:49:20.508 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-websocket (1.2.3-1) ... 2023-12-10T18:49:20.845 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-kubernetes. 2023-12-10T18:49:20.873 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2023-12-10T18:49:20.912 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2023-12-10T18:49:21.305 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph-volume. 2023-12-10T18:49:21.321 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-130-g3d2db44f-1jammy_all.deb ... 2023-12-10T18:49:21.354 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph-volume (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:49:21.603 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-12-10T18:49:21.617 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libcephfs-dev. 2023-12-10T18:49:21.619 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-130-g3d2db44f-1jammy_all.deb ... 2023-12-10T18:49:21.645 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-10T18:49:21.661 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:49:21.675 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libcephfs-dev (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:49:21.947 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package libonig5:amd64. 2023-12-10T18:49:21.961 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package lua-socket:amd64. 2023-12-10T18:49:21.974 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2023-12-10T18:49:21.989 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2023-12-10T18:49:22.013 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2023-12-10T18:49:22.027 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2023-12-10T18:49:22.341 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package libjq1:amd64. 2023-12-10T18:49:22.357 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package lua-sec:amd64. 2023-12-10T18:49:22.369 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2023-12-10T18:49:22.385 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2023-12-10T18:49:22.416 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-12-10T18:49:22.423 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2023-12-10T18:49:22.710 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package jq. 2023-12-10T18:49:22.726 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2023-12-10T18:49:22.768 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2023-12-10T18:49:22.769 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package nvme-cli. 2023-12-10T18:49:22.798 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2023-12-10T18:49:22.836 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2023-12-10T18:49:23.021 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package socat. 2023-12-10T18:49:23.037 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2023-12-10T18:49:23.079 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2023-12-10T18:49:23.199 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package pkg-config. 2023-12-10T18:49:23.227 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2023-12-10T18:49:23.257 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2023-12-10T18:49:23.373 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package xmlstarlet. 2023-12-10T18:49:23.389 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2023-12-10T18:49:23.423 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2023-12-10T18:49:23.493 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python-asyncssh-doc. 2023-12-10T18:49:23.521 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2023-12-10T18:49:23.560 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2023-12-10T18:49:23.786 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package ceph-test. 2023-12-10T18:49:23.814 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../097-ceph-test_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-10T18:49:23.852 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking ceph-test (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:49:23.955 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-iniconfig. 2023-12-10T18:49:23.983 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2023-12-10T18:49:24.021 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2023-12-10T18:49:24.345 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-pastescript. 2023-12-10T18:49:24.374 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2023-12-10T18:49:24.415 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-12-10T18:49:24.735 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-pluggy. 2023-12-10T18:49:24.763 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2023-12-10T18:49:24.801 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2023-12-10T18:49:25.113 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-psutil. 2023-12-10T18:49:25.141 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2023-12-10T18:49:25.179 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2023-12-10T18:49:25.524 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-py. 2023-12-10T18:49:25.552 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2023-12-10T18:49:25.590 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-py (1.10.0-1) ... 2023-12-10T18:49:25.919 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-pygments. 2023-12-10T18:49:25.947 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2023-12-10T18:49:25.985 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2023-12-10T18:49:26.414 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package ceph-volume. 2023-12-10T18:49:26.442 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-130-g3d2db44f-1jammy_all.deb ... 2023-12-10T18:49:26.464 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-pyinotify. 2023-12-10T18:49:26.481 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking ceph-volume (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:49:26.492 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2023-12-10T18:49:26.530 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2023-12-10T18:49:26.784 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package libcephfs-dev. 2023-12-10T18:49:26.801 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-10T18:49:26.825 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-toml. 2023-12-10T18:49:26.843 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking libcephfs-dev (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:49:26.853 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2023-12-10T18:49:26.891 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-toml (0.10.2-1) ... 2023-12-10T18:49:27.162 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package lua-socket:amd64. 2023-12-10T18:49:27.178 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-pytest. 2023-12-10T18:49:27.190 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2023-12-10T18:49:27.206 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2023-12-10T18:49:27.228 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2023-12-10T18:49:27.235 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2023-12-10T18:49:27.566 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package lua-sec:amd64. 2023-12-10T18:49:27.572 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-simplejson. 2023-12-10T18:49:27.594 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2023-12-10T18:49:27.601 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2023-12-10T18:49:27.624 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2023-12-10T18:49:27.639 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2023-12-10T18:49:27.942 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-12-10T18:49:27.970 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2023-12-10T18:49:28.008 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2023-12-10T18:49:28.011 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package nvme-cli. 2023-12-10T18:49:28.039 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2023-12-10T18:49:28.078 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2023-12-10T18:49:28.474 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package pkg-config. 2023-12-10T18:49:28.502 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2023-12-10T18:49:28.540 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2023-12-10T18:49:28.712 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package radosgw. 2023-12-10T18:49:28.740 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../116-radosgw_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-10T18:49:28.770 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking radosgw (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:49:28.793 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python-asyncssh-doc. 2023-12-10T18:49:28.821 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2023-12-10T18:49:28.860 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2023-12-10T18:49:29.329 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-iniconfig. 2023-12-10T18:49:29.357 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2023-12-10T18:49:29.396 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2023-12-10T18:49:29.498 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package rbd-fuse. 2023-12-10T18:49:29.515 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-10T18:49:29.556 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking rbd-fuse (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:49:29.698 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-pastescript. 2023-12-10T18:49:29.726 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2023-12-10T18:49:29.765 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-12-10T18:49:29.892 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package rbd-mirror. 2023-12-10T18:49:29.921 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../118-rbd-mirror_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-10T18:49:29.959 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking rbd-mirror (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:49:30.060 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-pluggy. 2023-12-10T18:49:30.076 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2023-12-10T18:49:30.110 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2023-12-10T18:49:30.404 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-psutil. 2023-12-10T18:49:30.414 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2023-12-10T18:49:30.431 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2023-12-10T18:49:30.470 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2023-12-10T18:49:30.656 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2023-12-10T18:49:30.782 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2023-12-10T18:49:30.806 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-py. 2023-12-10T18:49:30.834 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2023-12-10T18:49:30.873 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-py (1.10.0-1) ... 2023-12-10T18:49:31.052 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2023-12-10T18:49:31.176 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-pygments. 2023-12-10T18:49:31.204 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2023-12-10T18:49:31.243 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2023-12-10T18:49:31.394 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2023-12-10T18:49:31.713 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-pyinotify. 2023-12-10T18:49:31.741 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2023-12-10T18:49:31.779 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2023-12-10T18:49:31.813 INFO:teuthology.orchestra.run.smithi033.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2023-12-10T18:49:31.883 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up cephadm (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:49:32.065 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-toml. 2023-12-10T18:49:32.093 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2023-12-10T18:49:32.123 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-toml (0.10.2-1) ... 2023-12-10T18:49:32.368 INFO:teuthology.orchestra.run.smithi033.stdout:Adding system user cephadm....done 2023-12-10T18:49:32.426 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-pytest. 2023-12-10T18:49:32.455 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2023-12-10T18:49:32.501 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2023-12-10T18:49:32.523 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2023-12-10T18:49:32.824 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2023-12-10T18:49:32.854 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-simplejson. 2023-12-10T18:49:32.882 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2023-12-10T18:49:32.921 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2023-12-10T18:49:33.092 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2023-12-10T18:49:33.190 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-12-10T18:49:33.192 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2023-12-10T18:49:33.206 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2023-12-10T18:49:33.248 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2023-12-10T18:49:33.436 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-12-10T18:49:33.695 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2023-12-10T18:49:33.821 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-py (1.10.0-1) ... 2023-12-10T18:49:34.044 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package radosgw. 2023-12-10T18:49:34.071 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../116-radosgw_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-10T18:49:34.110 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking radosgw (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:49:34.131 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2023-12-10T18:49:34.499 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-cachetools (5.0.0-1) ... 2023-12-10T18:49:34.759 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-openssl (21.0.0-1) ... 2023-12-10T18:49:34.855 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package rbd-fuse. 2023-12-10T18:49:34.870 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-10T18:49:34.912 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking rbd-fuse (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:49:35.026 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up unzip (6.0-26ubuntu3.1) ... 2023-12-10T18:49:35.249 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package rbd-mirror. 2023-12-10T18:49:35.270 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2023-12-10T18:49:35.276 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../118-rbd-mirror_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-10T18:49:35.315 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking rbd-mirror (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:49:35.521 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2023-12-10T18:49:35.761 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2023-12-10T18:49:35.772 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2023-12-10T18:49:36.023 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-ceph-argparse (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:49:36.040 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2023-12-10T18:49:36.171 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2023-12-10T18:49:36.332 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-12-10T18:49:36.430 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2023-12-10T18:49:36.458 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2023-12-10T18:49:36.559 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2023-12-10T18:49:36.661 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2023-12-10T18:49:36.778 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2023-12-10T18:49:36.903 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2023-12-10T18:49:37.224 INFO:teuthology.orchestra.run.smithi076.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2023-12-10T18:49:37.282 INFO:teuthology.orchestra.run.smithi033.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2023-12-10T18:49:37.289 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up cephadm (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:49:37.309 INFO:teuthology.orchestra.run.smithi033.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2023-12-10T18:49:37.392 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-12-10T18:49:37.516 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2023-12-10T18:49:37.740 INFO:teuthology.orchestra.run.smithi076.stdout:Adding system user cephadm....done 2023-12-10T18:49:37.895 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2023-12-10T18:49:37.900 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-12-10T18:49:38.001 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-natsort (8.0.2-1) ... 2023-12-10T18:49:38.163 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2023-12-10T18:49:38.254 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2023-12-10T18:49:38.397 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2023-12-10T18:49:38.515 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2023-12-10T18:49:38.522 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2023-12-10T18:49:38.749 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-12-10T18:49:38.840 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up zip (3.0-12build2) ... 2023-12-10T18:49:38.966 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2023-12-10T18:49:39.000 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2023-12-10T18:49:39.135 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-py (1.10.0-1) ... 2023-12-10T18:49:39.469 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2023-12-10T18:49:39.668 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2023-12-10T18:49:39.830 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-cachetools (5.0.0-1) ... 2023-12-10T18:49:39.921 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2023-12-10T18:49:40.020 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2023-12-10T18:49:40.072 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-openssl (21.0.0-1) ... 2023-12-10T18:49:40.138 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2023-12-10T18:49:40.315 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up unzip (6.0-26ubuntu3.1) ... 2023-12-10T18:49:40.430 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2023-12-10T18:49:40.600 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2023-12-10T18:49:40.832 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-certifi (2020.6.20-1) ... 2023-12-10T18:49:40.851 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2023-12-10T18:49:41.082 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2023-12-10T18:49:41.110 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2023-12-10T18:49:41.386 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-ceph-argparse (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:49:41.451 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-12-10T18:49:41.649 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-12-10T18:49:41.772 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2023-12-10T18:49:41.777 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-12-10T18:49:41.897 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2023-12-10T18:49:42.007 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2023-12-10T18:49:42.120 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2023-12-10T18:49:42.241 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2023-12-10T18:49:42.379 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2023-12-10T18:49:42.506 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-ceph-common (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:49:42.654 INFO:teuthology.orchestra.run.smithi076.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2023-12-10T18:49:42.713 INFO:teuthology.orchestra.run.smithi076.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2023-12-10T18:49:42.769 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-12-10T18:49:42.807 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-idna (3.3-1) ... 2023-12-10T18:49:42.895 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2023-12-10T18:49:43.125 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2023-12-10T18:49:43.254 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-12-10T18:49:43.380 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-natsort (8.0.2-1) ... 2023-12-10T18:49:43.656 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2023-12-10T18:49:43.899 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2023-12-10T18:49:44.200 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up zip (3.0-12build2) ... 2023-12-10T18:49:44.297 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2023-12-10T18:49:44.309 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2023-12-10T18:49:44.549 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-10T18:49:44.683 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-toml (0.10.2-1) ... 2023-12-10T18:49:44.942 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2023-12-10T18:49:44.953 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2023-12-10T18:49:45.077 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2023-12-10T18:49:45.202 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2023-12-10T18:49:45.212 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2023-12-10T18:49:45.338 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2023-12-10T18:49:45.464 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2023-12-10T18:49:45.496 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2023-12-10T18:49:45.763 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2023-12-10T18:49:45.801 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2023-12-10T18:49:46.039 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-10T18:49:46.191 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2023-12-10T18:49:46.433 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-certifi (2020.6.20-1) ... 2023-12-10T18:49:46.501 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2023-12-10T18:49:46.676 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2023-12-10T18:49:46.760 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-logutils (0.3.3-8) ... 2023-12-10T18:49:47.019 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-tempora (4.1.2-1) ... 2023-12-10T18:49:47.052 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-12-10T18:49:47.279 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-12-10T18:49:47.353 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-12-10T18:49:47.530 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-prettytable (2.5.0-2) ... 2023-12-10T18:49:47.688 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2023-12-10T18:49:47.797 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-12-10T18:49:47.924 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-websocket (1.2.3-1) ... 2023-12-10T18:49:47.939 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2023-12-10T18:49:48.065 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-ceph-common (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:49:48.208 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2023-12-10T18:49:48.334 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-12-10T18:49:48.358 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-idna (3.3-1) ... 2023-12-10T18:49:48.627 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2023-12-10T18:49:48.684 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2023-12-10T18:49:48.928 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2023-12-10T18:49:49.179 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2023-12-10T18:49:49.305 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2023-12-10T18:49:49.431 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2023-12-10T18:49:49.520 INFO:teuthology.orchestra.run.smithi033.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2023-12-10T18:49:49.783 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-12-10T18:49:49.847 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2023-12-10T18:49:49.884 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2023-12-10T18:49:50.073 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-10T18:49:50.183 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-toml (0.10.2-1) ... 2023-12-10T18:49:50.252 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2023-12-10T18:49:50.442 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2023-12-10T18:49:50.495 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up lua-any (27ubuntu1) ... 2023-12-10T18:49:50.569 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2023-12-10T18:49:50.596 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-portend (3.0.0-1) ... 2023-12-10T18:49:50.694 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2023-12-10T18:49:50.855 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-10T18:49:50.990 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-jinja2 (3.0.3-1) ... 2023-12-10T18:49:51.012 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2023-12-10T18:49:51.255 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2023-12-10T18:49:51.307 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2023-12-10T18:49:51.497 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-10T18:49:51.608 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-12-10T18:49:51.615 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2023-12-10T18:49:51.885 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2023-12-10T18:49:51.942 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2023-12-10T18:49:51.985 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-rsa (4.8-1) ... 2023-12-10T18:49:52.176 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-logutils (0.3.3-8) ... 2023-12-10T18:49:52.245 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-webtest (2.0.35-1) ... 2023-12-10T18:49:52.452 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-tempora (4.1.2-1) ... 2023-12-10T18:49:52.530 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-12-10T18:49:52.711 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-12-10T18:49:52.874 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2023-12-10T18:49:52.971 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-prettytable (2.5.0-2) ... 2023-12-10T18:49:53.238 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-12-10T18:49:53.259 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-12-10T18:49:53.339 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-websocket (1.2.3-1) ... 2023-12-10T18:49:53.535 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2023-12-10T18:49:53.607 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2023-12-10T18:49:53.708 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-12-10T18:49:53.888 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-12-10T18:49:53.984 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2023-12-10T18:49:54.014 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up librados2 (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:49:54.140 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libsqlite3-mod-ceph (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:49:54.266 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2023-12-10T18:49:54.285 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2023-12-10T18:49:54.535 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libcephfs2 (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:49:54.536 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2023-12-10T18:49:54.644 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libradosstriper1 (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:49:54.662 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2023-12-10T18:49:54.771 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2023-12-10T18:49:54.799 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-google-auth (1.5.1-3) ... 2023-12-10T18:49:54.852 INFO:teuthology.orchestra.run.smithi076.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2023-12-10T18:49:55.081 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-12-10T18:49:55.097 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up librbd1 (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:49:55.182 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2023-12-10T18:49:55.223 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph-mgr-modules-core (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:49:55.349 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph-fuse (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:49:55.558 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2023-12-10T18:49:55.568 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-12-10T18:49:55.569 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-12-10T18:49:55.818 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up lua-any (27ubuntu1) ... 2023-12-10T18:49:55.944 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-portend (3.0.0-1) ... 2023-12-10T18:49:56.159 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libcephfs-dev (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:49:56.203 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-10T18:49:56.260 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-rados (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:49:56.329 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-jinja2 (3.0.3-1) ... 2023-12-10T18:49:56.361 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up librgw2 (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:49:56.479 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2023-12-10T18:49:56.913 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2023-12-10T18:49:57.223 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-12-10T18:49:57.482 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2023-12-10T18:49:57.583 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-rsa (4.8-1) ... 2023-12-10T18:49:57.774 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-rbd (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:49:57.842 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-webtest (2.0.35-1) ... 2023-12-10T18:49:57.900 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up rbd-fuse (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:49:58.026 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-rgw (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:49:58.110 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-12-10T18:49:58.154 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-cephfs (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:49:58.278 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph-common (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:49:58.455 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2023-12-10T18:49:58.812 INFO:teuthology.orchestra.run.smithi033.stdout:Adding group ceph....done 2023-12-10T18:49:58.831 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-12-10T18:49:59.116 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2023-12-10T18:49:59.172 INFO:teuthology.orchestra.run.smithi033.stdout:Adding system user ceph....done 2023-12-10T18:49:59.299 INFO:teuthology.orchestra.run.smithi033.stdout:Setting system user ceph properties....done 2023-12-10T18:49:59.318 INFO:teuthology.orchestra.run.smithi033.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-12-10T18:49:59.435 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-12-10T18:49:59.463 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-12-10T18:49:59.561 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up librados2 (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:49:59.712 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up libsqlite3-mod-ceph (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:49:59.803 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-12-10T18:49:59.838 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2023-12-10T18:50:00.207 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up libcephfs2 (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:50:00.358 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up libradosstriper1 (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:50:00.379 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph-test (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:50:00.514 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up radosgw (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:50:00.517 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-google-auth (1.5.1-3) ... 2023-12-10T18:50:00.828 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up librbd1 (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:50:00.953 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up ceph-mgr-modules-core (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:50:00.984 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-12-10T18:50:00.984 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-12-10T18:50:01.079 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up ceph-fuse (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:50:01.289 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-12-10T18:50:01.289 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-12-10T18:50:01.441 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph-base (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:50:01.772 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up libcephfs-dev (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:50:01.776 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-12-10T18:50:01.940 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-rados (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:50:02.074 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up librgw2 (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:50:02.225 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph-mds (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:50:02.226 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2023-12-10T18:50:02.434 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-12-10T18:50:02.434 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-12-10T18:50:02.894 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up rbd-mirror (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:50:03.171 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2023-12-10T18:50:03.171 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2023-12-10T18:50:03.472 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-rbd (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:50:03.597 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up rbd-fuse (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:50:03.698 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-rgw (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:50:03.746 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph-mgr (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:50:03.799 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-cephfs (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:50:03.909 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up ceph-common (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:50:03.982 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-12-10T18:50:03.982 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-12-10T18:50:04.418 INFO:teuthology.orchestra.run.smithi076.stdout:Adding group ceph....done 2023-12-10T18:50:04.464 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph-osd (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:50:04.748 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-12-10T18:50:04.748 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-12-10T18:50:04.777 INFO:teuthology.orchestra.run.smithi076.stdout:Adding system user ceph....done 2023-12-10T18:50:04.955 INFO:teuthology.orchestra.run.smithi076.stdout:Setting system user ceph properties....done 2023-12-10T18:50:04.967 INFO:teuthology.orchestra.run.smithi076.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-12-10T18:50:05.101 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-12-10T18:50:05.241 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph-mgr-k8sevents (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:50:05.342 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:50:05.425 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-12-10T18:50:05.468 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph-mon (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:50:05.676 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-12-10T18:50:05.676 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-12-10T18:50:05.926 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up ceph-test (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:50:06.052 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up radosgw (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:50:06.119 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph-mgr-cephadm (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:50:06.271 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:50:06.423 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph-mgr-dashboard (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:50:06.513 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-12-10T18:50:06.513 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-12-10T18:50:06.582 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph-volume (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:50:06.927 INFO:teuthology.orchestra.run.smithi033.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2023-12-10T18:50:06.962 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up ceph-base (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:50:07.028 INFO:teuthology.orchestra.run.smithi033.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-12-10T18:50:07.164 INFO:teuthology.orchestra.run.smithi033.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-12-10T18:50:07.313 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-12-10T18:50:07.806 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up ceph-mds (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:50:08.018 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-12-10T18:50:08.018 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-12-10T18:50:08.289 INFO:teuthology.orchestra.run.smithi033.stdout:Processing triggers for install-info (6.8-4build1) ... 2023-12-10T18:50:08.490 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up rbd-mirror (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:50:08.680 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2023-12-10T18:50:08.680 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2023-12-10T18:50:08.959 INFO:teuthology.orchestra.run.smithi033.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-12-10T18:50:08.963 DEBUG:teuthology.parallel:result is None 2023-12-10T18:50:09.133 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up ceph-mgr (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:50:09.356 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-12-10T18:50:09.357 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-12-10T18:50:09.826 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up ceph-osd (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:50:10.097 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-12-10T18:50:10.097 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-12-10T18:50:10.578 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up ceph-mgr-k8sevents (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:50:10.704 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:50:10.855 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up ceph-mon (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:50:11.057 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-12-10T18:50:11.058 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-12-10T18:50:11.555 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up ceph-mgr-cephadm (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:50:11.790 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up ceph (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:50:11.908 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up ceph-mgr-dashboard (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:50:12.067 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up ceph-volume (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-10T18:50:12.572 INFO:teuthology.orchestra.run.smithi076.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2023-12-10T18:50:12.673 INFO:teuthology.orchestra.run.smithi076.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-12-10T18:50:12.817 INFO:teuthology.orchestra.run.smithi076.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-12-10T18:50:13.846 INFO:teuthology.orchestra.run.smithi076.stdout:Processing triggers for install-info (6.8-4build1) ... 2023-12-10T18:50:14.492 INFO:teuthology.orchestra.run.smithi076.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-12-10T18:50:14.494 DEBUG:teuthology.parallel:result is None 2023-12-10T18:50:14.495 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=3d2db44fa991bc355074aa9b7647f63b77201fef 2023-12-10T18:50:14.826 DEBUG:teuthology.orchestra.run.smithi033:> dpkg-query -W -f '${Version}' ceph 2023-12-10T18:50:14.846 INFO:teuthology.orchestra.run.smithi033.stdout:19.0.0-130-g3d2db44f-1jammy 2023-12-10T18:50:14.847 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-130-g3d2db44f-1jammy 2023-12-10T18:50:14.847 INFO:teuthology.task.install:The correct ceph version 19.0.0-130-g3d2db44f-1jammy is installed. 2023-12-10T18:50:14.847 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=3d2db44fa991bc355074aa9b7647f63b77201fef 2023-12-10T18:50:15.154 DEBUG:teuthology.orchestra.run.smithi076:> dpkg-query -W -f '${Version}' ceph 2023-12-10T18:50:15.173 INFO:teuthology.orchestra.run.smithi076.stdout:19.0.0-130-g3d2db44f-1jammy 2023-12-10T18:50:15.174 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-130-g3d2db44f-1jammy 2023-12-10T18:50:15.174 INFO:teuthology.task.install:The correct ceph version 19.0.0-130-g3d2db44f-1jammy is installed. 2023-12-10T18:50:15.174 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-12-10T18:50:15.174 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-10T18:50:15.174 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-10T18:50:15.189 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2023-12-10T18:50:15.189 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-10T18:50:15.231 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-12-10T18:50:15.231 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-10T18:50:15.231 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/usr/bin/daemon-helper 2023-12-10T18:50:15.244 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-10T18:50:15.299 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2023-12-10T18:50:15.299 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/usr/bin/daemon-helper 2023-12-10T18:50:15.313 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-10T18:50:15.367 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-12-10T18:50:15.367 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-10T18:50:15.367 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-10T18:50:15.381 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-10T18:50:15.435 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2023-12-10T18:50:15.435 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-10T18:50:15.448 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-10T18:50:15.502 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-12-10T18:50:15.502 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-10T18:50:15.502 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/usr/bin/stdin-killer 2023-12-10T18:50:15.515 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-10T18:50:15.570 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2023-12-10T18:50:15.570 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/usr/bin/stdin-killer 2023-12-10T18:50:15.584 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-10T18:50:15.638 INFO:teuthology.run_tasks:Running task ceph... 2023-12-10T18:50:15.707 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-12-10T18:50:15.708 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 777 /var/log/ceph 2023-12-10T18:50:15.710 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod 777 /var/log/ceph 2023-12-10T18:50:15.725 INFO:tasks.ceph:Disabling ceph logrotate... 2023-12-10T18:50:15.725 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -f -- /etc/logrotate.d/ceph 2023-12-10T18:50:15.764 DEBUG:teuthology.orchestra.run.smithi076:> sudo rm -f -- /etc/logrotate.d/ceph 2023-12-10T18:50:15.778 INFO:tasks.ceph:Creating extra log directories... 2023-12-10T18:50:15.778 DEBUG:teuthology.orchestra.run.smithi033:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-12-10T18:50:15.820 DEBUG:teuthology.orchestra.run.smithi076:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-12-10T18:50:15.851 INFO:tasks.ceph:Creating ceph cluster cluster1... 2023-12-10T18:50:15.852 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-12-10T18:50:15.852 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/dis-2023-12-10_16:10:42-rbd-wip-dis-testing-1-distro-default-smithi/7486362', 'branch': 'wip-dis-testing-1', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-stupid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-stress-workunit-minimum}', 'email': 'idryomov@gmail.com', 'first_in_suite': False, 'job_id': '7486362', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'dis-2023-12-10_16:10:42-rbd-wip-dis-testing-1-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-dis-testing-1'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '3d2db44fa991bc355074aa9b7647f63b77201fef'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '3d2db44fa991bc355074aa9b7647f63b77201fef'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-dis-testing-1', 'sha1': '3d2db44fa991bc355074aa9b7647f63b77201fef'}}, 'owner': 'scheduled_dis@teuthology', 'priority': 74, 'repo': 'https://github.com/ceph/ceph-ci', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 7033, 'sha1': '3d2db44fa991bc355074aa9b7647f63b77201fef', 'sleep_before_teardown': 0, 'subset': '91/99', 'suite': 'rbd', 'suite_branch': 'wip-dis-testing-1', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci', 'suite_sha1': '3d2db44fa991bc355074aa9b7647f63b77201fef', 'targets': {'smithi033.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCg4fpIf9BgAE0K5vY3eaCYNnAdpFkPYg83UNzpXJx+VdnMglbkcDNn9oNHV0h8GlQ68zOzCcQCAjDqUW/H3wNI=', 'smithi076.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDz6mTNSmU7E4IwEUFsmyAfIf0TTvzaqidmq2OKROs5pSoYL6ng2SXq1yuh1NB7AMqvNCBdduJ5kk6F5IVYNexs='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': '3d2db44fa991bc355074aa9b7647f63b77201fef'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '3d2db44fa991bc355074aa9b7647f63b77201fef'}}, {'ceph': {'cluster': 'cluster2'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_MODE': 'snapshot', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'f2de108beb0a6dde7f3e6c96039071828a1f6269', 'timestamp': '2023-12-10_16:10:42', 'tube': 'smithi', 'user': 'dis', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.232157'} 2023-12-10T18:50:15.852 DEBUG:teuthology.orchestra.run.smithi033:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2023-12-10T18:50:15.876 DEBUG:teuthology.orchestra.run.smithi076:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2023-12-10T18:50:15.886 DEBUG:teuthology.orchestra.run.smithi033:> sudo install -d -m0777 -- /var/run/ceph 2023-12-10T18:50:15.924 DEBUG:teuthology.orchestra.run.smithi076:> sudo install -d -m0777 -- /var/run/ceph 2023-12-10T18:50:15.942 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-10T18:50:15.942 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/scratch_devs of=/dev/stdout 2023-12-10T18:50:15.983 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-10T18:50:15.983 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_1 2023-12-10T18:50:16.035 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-10T18:50:16.035 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-10T18:50:16.036 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 5h/5d Inode: 886 Links: 1 2023-12-10T18:50:16.036 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-10T18:50:16.036 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2023-12-10 18:44:55.325199285 +0000 2023-12-10T18:50:16.036 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2023-12-10 18:44:54.957202568 +0000 2023-12-10T18:50:16.036 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2023-12-10 18:44:54.957202568 +0000 2023-12-10T18:50:16.036 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: - 2023-12-10T18:50:16.036 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-10T18:50:16.090 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2023-12-10T18:50:16.090 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2023-12-10T18:50:16.090 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.00044352 s, 1.2 MB/s 2023-12-10T18:50:16.091 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-10T18:50:16.140 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_2 2023-12-10T18:50:16.187 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-10T18:50:16.188 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-10T18:50:16.188 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 5h/5d Inode: 899 Links: 1 2023-12-10T18:50:16.188 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-10T18:50:16.188 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2023-12-10 18:44:55.993193325 +0000 2023-12-10T18:50:16.188 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2023-12-10 18:44:55.673196180 +0000 2023-12-10T18:50:16.188 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2023-12-10 18:44:55.673196180 +0000 2023-12-10T18:50:16.188 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: - 2023-12-10T18:50:16.188 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-10T18:50:16.241 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2023-12-10T18:50:16.242 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2023-12-10T18:50:16.242 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.000418774 s, 1.2 MB/s 2023-12-10T18:50:16.243 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-10T18:50:16.292 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_3 2023-12-10T18:50:16.339 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-10T18:50:16.339 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-10T18:50:16.339 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 5h/5d Inode: 918 Links: 1 2023-12-10T18:50:16.339 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-10T18:50:16.339 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2023-12-10 18:44:56.593187973 +0000 2023-12-10T18:50:16.339 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2023-12-10 18:44:56.237191149 +0000 2023-12-10T18:50:16.339 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2023-12-10 18:44:56.237191149 +0000 2023-12-10T18:50:16.339 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: - 2023-12-10T18:50:16.340 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-10T18:50:16.393 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2023-12-10T18:50:16.393 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2023-12-10T18:50:16.394 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.000394279 s, 1.3 MB/s 2023-12-10T18:50:16.395 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-10T18:50:16.444 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_4 2023-12-10T18:50:16.491 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-10T18:50:16.491 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-10T18:50:16.491 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 5h/5d Inode: 934 Links: 1 2023-12-10T18:50:16.491 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-10T18:50:16.491 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2023-12-10 18:44:57.161182906 +0000 2023-12-10T18:50:16.491 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2023-12-10 18:44:56.813186011 +0000 2023-12-10T18:50:16.491 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2023-12-10 18:44:56.813186011 +0000 2023-12-10T18:50:16.492 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: - 2023-12-10T18:50:16.492 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-10T18:50:16.545 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2023-12-10T18:50:16.545 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2023-12-10T18:50:16.545 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.000389003 s, 1.3 MB/s 2023-12-10T18:50:16.546 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-10T18:50:16.596 INFO:tasks.ceph:osd dev map: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-10T18:50:16.597 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi033.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2023-12-10T18:50:16.597 INFO:tasks.ceph:Generating config... 2023-12-10T18:50:16.598 INFO:tasks.ceph:[client] debug rbd = 20 2023-12-10T18:50:16.598 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2023-12-10T18:50:16.598 INFO:tasks.ceph:[client] log to stderr = False 2023-12-10T18:50:16.598 INFO:tasks.ceph:[client] rbd default features = 125 2023-12-10T18:50:16.598 INFO:tasks.ceph:[client] rbd mirror image policy type = simple 2023-12-10T18:50:16.598 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-10T18:50:16.599 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-10T18:50:16.599 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-10T18:50:16.599 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-10T18:50:16.599 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-10T18:50:16.599 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-10T18:50:16.599 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-10T18:50:16.599 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-10T18:50:16.599 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-12-10T18:50:16.599 INFO:tasks.ceph:[global] mon warn on pool no app = False 2023-12-10T18:50:16.599 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-12-10T18:50:16.599 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-12-10T18:50:16.599 INFO:tasks.ceph:[mgr] debug ms = 1 2023-12-10T18:50:16.599 INFO:tasks.ceph:[mon] debug mon = 20 2023-12-10T18:50:16.599 INFO:tasks.ceph:[mon] debug ms = 1 2023-12-10T18:50:16.600 INFO:tasks.ceph:[mon] debug paxos = 20 2023-12-10T18:50:16.600 INFO:tasks.ceph:[osd] bdev async discard = True 2023-12-10T18:50:16.600 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-12-10T18:50:16.600 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2023-12-10T18:50:16.600 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-12-10T18:50:16.600 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-12-10T18:50:16.600 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-12-10T18:50:16.600 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-12-10T18:50:16.600 INFO:tasks.ceph:[osd] debug ms = 1 2023-12-10T18:50:16.600 INFO:tasks.ceph:[osd] debug osd = 20 2023-12-10T18:50:16.600 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-12-10T18:50:16.600 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-12-10T18:50:16.600 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-12-10T18:50:16.600 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-12-10T18:50:16.600 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-12-10T18:50:16.601 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-12-10T18:50:16.601 INFO:tasks.ceph:Setting up cluster1.mon.a... 2023-12-10T18:50:16.601 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster1.keyring 2023-12-10T18:50:16.672 INFO:teuthology.orchestra.run.smithi033.stdout:creating /etc/ceph/cluster1.keyring 2023-12-10T18:50:16.680 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster1.keyring 2023-12-10T18:50:16.756 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2023-12-10T18:50:16.811 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster1.mon.a', '172.21.15.33')] 2023-12-10T18:50:16.811 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.33', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2023-12-10T18:50:16.812 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-10T18:50:16.812 DEBUG:teuthology.orchestra.run.smithi033:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-12-10T18:50:16.859 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.33 --print /home/ubuntu/cephtest/cluster1.monmap 2023-12-10T18:50:16.939 INFO:teuthology.orchestra.run.smithi033.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster1.monmap 2023-12-10T18:50:16.939 INFO:teuthology.orchestra.run.smithi033.stdout:monmaptool: generated fsid d3c37dcf-9605-4f22-911b-14b903774172 2023-12-10T18:50:16.939 INFO:teuthology.orchestra.run.smithi033.stdout:setting min_mon_release = pacific 2023-12-10T18:50:16.939 INFO:teuthology.orchestra.run.smithi033.stdout:epoch 0 2023-12-10T18:50:16.939 INFO:teuthology.orchestra.run.smithi033.stdout:fsid d3c37dcf-9605-4f22-911b-14b903774172 2023-12-10T18:50:16.939 INFO:teuthology.orchestra.run.smithi033.stdout:last_changed 2023-12-10T18:50:16.939509+0000 2023-12-10T18:50:16.939 INFO:teuthology.orchestra.run.smithi033.stdout:created 2023-12-10T18:50:16.939509+0000 2023-12-10T18:50:16.939 INFO:teuthology.orchestra.run.smithi033.stdout:min_mon_release 16 (pacific) 2023-12-10T18:50:16.939 INFO:teuthology.orchestra.run.smithi033.stdout:election_strategy: 1 2023-12-10T18:50:16.939 INFO:teuthology.orchestra.run.smithi033.stdout:0: [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] mon.a 2023-12-10T18:50:16.939 INFO:teuthology.orchestra.run.smithi033.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster1.monmap (1 monitors) 2023-12-10T18:50:16.941 DEBUG:teuthology.orchestra.run.smithi033:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-12-10T18:50:16.987 INFO:tasks.ceph:Writing /etc/ceph/cluster1.conf for FSID d3c37dcf-9605-4f22-911b-14b903774172... 2023-12-10T18:50:16.988 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2023-12-10T18:50:17.032 DEBUG:teuthology.orchestra.run.smithi076:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2023-12-10T18:50:17.061 INFO:teuthology.orchestra.run.smithi033.stdout:[global] 2023-12-10T18:50:17.061 INFO:teuthology.orchestra.run.smithi033.stdout: chdir = "" 2023-12-10T18:50:17.061 INFO:teuthology.orchestra.run.smithi033.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-10T18:50:17.061 INFO:teuthology.orchestra.run.smithi033.stdout: auth supported = cephx 2023-12-10T18:50:17.061 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-10T18:50:17.061 INFO:teuthology.orchestra.run.smithi033.stdout: filestore xattr use omap = true 2023-12-10T18:50:17.061 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-10T18:50:17.061 INFO:teuthology.orchestra.run.smithi033.stdout: mon clock drift allowed = 1.000 2023-12-10T18:50:17.062 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-10T18:50:17.062 INFO:teuthology.orchestra.run.smithi033.stdout: osd crush chooseleaf type = 0 2023-12-10T18:50:17.062 INFO:teuthology.orchestra.run.smithi033.stdout: auth debug = true 2023-12-10T18:50:17.062 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-10T18:50:17.062 INFO:teuthology.orchestra.run.smithi033.stdout: ms die on old message = true 2023-12-10T18:50:17.062 INFO:teuthology.orchestra.run.smithi033.stdout: ms die on bug = true 2023-12-10T18:50:17.062 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-10T18:50:17.062 INFO:teuthology.orchestra.run.smithi033.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-10T18:50:17.062 INFO:teuthology.orchestra.run.smithi033.stdout: mon pg warn max object skew = 0 2023-12-10T18:50:17.062 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-10T18:50:17.062 INFO:teuthology.orchestra.run.smithi033.stdout: # disable pg_autoscaler by default for new pools 2023-12-10T18:50:17.062 INFO:teuthology.orchestra.run.smithi033.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-10T18:50:17.062 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-10T18:50:17.062 INFO:teuthology.orchestra.run.smithi033.stdout: osd pool default size = 2 2023-12-10T18:50:17.063 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-10T18:50:17.063 INFO:teuthology.orchestra.run.smithi033.stdout: mon osd allow primary affinity = true 2023-12-10T18:50:17.063 INFO:teuthology.orchestra.run.smithi033.stdout: mon osd allow pg remap = true 2023-12-10T18:50:17.063 INFO:teuthology.orchestra.run.smithi033.stdout: mon warn on legacy crush tunables = false 2023-12-10T18:50:17.063 INFO:teuthology.orchestra.run.smithi033.stdout: mon warn on crush straw calc version zero = false 2023-12-10T18:50:17.063 INFO:teuthology.orchestra.run.smithi033.stdout: mon warn on no sortbitwise = false 2023-12-10T18:50:17.063 INFO:teuthology.orchestra.run.smithi033.stdout: mon warn on osd down out interval zero = false 2023-12-10T18:50:17.063 INFO:teuthology.orchestra.run.smithi033.stdout: mon warn on too few osds = false 2023-12-10T18:50:17.063 INFO:teuthology.orchestra.run.smithi033.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-10T18:50:17.063 INFO:teuthology.orchestra.run.smithi033.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-10T18:50:17.063 INFO:teuthology.orchestra.run.smithi033.stdout: mon_allow_pool_size_one = true 2023-12-10T18:50:17.063 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-10T18:50:17.063 INFO:teuthology.orchestra.run.smithi033.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-10T18:50:17.063 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-10T18:50:17.063 INFO:teuthology.orchestra.run.smithi033.stdout: osd default data pool replay window = 5 2023-12-10T18:50:17.063 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-10T18:50:17.064 INFO:teuthology.orchestra.run.smithi033.stdout: mon allow pool delete = true 2023-12-10T18:50:17.064 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-10T18:50:17.064 INFO:teuthology.orchestra.run.smithi033.stdout: mon cluster log file level = debug 2023-12-10T18:50:17.064 INFO:teuthology.orchestra.run.smithi033.stdout: debug asserts on shutdown = true 2023-12-10T18:50:17.064 INFO:teuthology.orchestra.run.smithi033.stdout: mon health detail to clog = false 2023-12-10T18:50:17.064 INFO:teuthology.orchestra.run.smithi033.stdout: mon host = 172.21.15.33 2023-12-10T18:50:17.064 INFO:teuthology.orchestra.run.smithi033.stdout: mon client directed command retry = 5 2023-12-10T18:50:17.064 INFO:teuthology.orchestra.run.smithi033.stdout: mon warn on pool no app = False 2023-12-10T18:50:17.064 INFO:teuthology.orchestra.run.smithi033.stdout: ms inject socket failures = 5000 2023-12-10T18:50:17.064 INFO:teuthology.orchestra.run.smithi033.stdout: fsid = d3c37dcf-9605-4f22-911b-14b903774172 2023-12-10T18:50:17.064 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-10T18:50:17.064 INFO:teuthology.orchestra.run.smithi033.stdout:[osd] 2023-12-10T18:50:17.064 INFO:teuthology.orchestra.run.smithi033.stdout: osd journal size = 100 2023-12-10T18:50:17.064 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-10T18:50:17.064 INFO:teuthology.orchestra.run.smithi033.stdout: osd scrub load threshold = 5.0 2023-12-10T18:50:17.065 INFO:teuthology.orchestra.run.smithi033.stdout: osd scrub max interval = 600 2023-12-10T18:50:17.065 INFO:teuthology.orchestra.run.smithi033.stdout: osd mclock profile = high_recovery_ops 2023-12-10T18:50:17.065 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-10T18:50:17.065 INFO:teuthology.orchestra.run.smithi033.stdout: osd recover clone overlap = true 2023-12-10T18:50:17.065 INFO:teuthology.orchestra.run.smithi033.stdout: osd recovery max chunk = 1048576 2023-12-10T18:50:17.065 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-10T18:50:17.065 INFO:teuthology.orchestra.run.smithi033.stdout: osd debug shutdown = true 2023-12-10T18:50:17.065 INFO:teuthology.orchestra.run.smithi033.stdout: osd debug op order = true 2023-12-10T18:50:17.065 INFO:teuthology.orchestra.run.smithi033.stdout: osd debug verify stray on activate = true 2023-12-10T18:50:17.065 INFO:teuthology.orchestra.run.smithi033.stdout: osd debug trim objects = true 2023-12-10T18:50:17.065 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-10T18:50:17.065 INFO:teuthology.orchestra.run.smithi033.stdout: osd open classes on start = true 2023-12-10T18:50:17.065 INFO:teuthology.orchestra.run.smithi033.stdout: osd debug pg log writeout = true 2023-12-10T18:50:17.065 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-10T18:50:17.065 INFO:teuthology.orchestra.run.smithi033.stdout: osd deep scrub update digest min age = 30 2023-12-10T18:50:17.065 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-10T18:50:17.065 INFO:teuthology.orchestra.run.smithi033.stdout: osd map max advance = 10 2023-12-10T18:50:17.066 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-10T18:50:17.066 INFO:teuthology.orchestra.run.smithi033.stdout: journal zero on create = true 2023-12-10T18:50:17.066 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-10T18:50:17.066 INFO:teuthology.orchestra.run.smithi033.stdout: filestore ondisk finisher threads = 3 2023-12-10T18:50:17.066 INFO:teuthology.orchestra.run.smithi033.stdout: filestore apply finisher threads = 3 2023-12-10T18:50:17.066 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-10T18:50:17.066 INFO:teuthology.orchestra.run.smithi033.stdout: bdev debug aio = true 2023-12-10T18:50:17.066 INFO:teuthology.orchestra.run.smithi033.stdout: osd debug misdirected ops = true 2023-12-10T18:50:17.066 INFO:teuthology.orchestra.run.smithi033.stdout: bdev async discard = True 2023-12-10T18:50:17.066 INFO:teuthology.orchestra.run.smithi033.stdout: bdev enable discard = True 2023-12-10T18:50:17.066 INFO:teuthology.orchestra.run.smithi033.stdout: bluestore allocator = stupid 2023-12-10T18:50:17.066 INFO:teuthology.orchestra.run.smithi033.stdout: bluestore block size = 96636764160 2023-12-10T18:50:17.066 INFO:teuthology.orchestra.run.smithi033.stdout: bluestore fsck on mount = True 2023-12-10T18:50:17.066 INFO:teuthology.orchestra.run.smithi033.stdout: debug bluefs = 1/20 2023-12-10T18:50:17.066 INFO:teuthology.orchestra.run.smithi033.stdout: debug bluestore = 1/20 2023-12-10T18:50:17.067 INFO:teuthology.orchestra.run.smithi033.stdout: debug ms = 1 2023-12-10T18:50:17.067 INFO:teuthology.orchestra.run.smithi033.stdout: debug osd = 20 2023-12-10T18:50:17.067 INFO:teuthology.orchestra.run.smithi033.stdout: debug rocksdb = 4/10 2023-12-10T18:50:17.067 INFO:teuthology.orchestra.run.smithi033.stdout: mon osd backfillfull_ratio = 0.85 2023-12-10T18:50:17.067 INFO:teuthology.orchestra.run.smithi033.stdout: mon osd full ratio = 0.9 2023-12-10T18:50:17.067 INFO:teuthology.orchestra.run.smithi033.stdout: mon osd nearfull ratio = 0.8 2023-12-10T18:50:17.067 INFO:teuthology.orchestra.run.smithi033.stdout: osd failsafe full ratio = 0.95 2023-12-10T18:50:17.067 INFO:teuthology.orchestra.run.smithi033.stdout: osd objectstore = bluestore 2023-12-10T18:50:17.067 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-10T18:50:17.067 INFO:teuthology.orchestra.run.smithi033.stdout:[mgr] 2023-12-10T18:50:17.067 INFO:teuthology.orchestra.run.smithi033.stdout: debug ms = 1 2023-12-10T18:50:17.067 INFO:teuthology.orchestra.run.smithi033.stdout: debug mgr = 20 2023-12-10T18:50:17.067 INFO:teuthology.orchestra.run.smithi033.stdout: debug mon = 20 2023-12-10T18:50:17.067 INFO:teuthology.orchestra.run.smithi033.stdout: debug auth = 20 2023-12-10T18:50:17.067 INFO:teuthology.orchestra.run.smithi033.stdout: mon reweight min pgs per osd = 4 2023-12-10T18:50:17.068 INFO:teuthology.orchestra.run.smithi033.stdout: mon reweight min bytes per osd = 10 2023-12-10T18:50:17.068 INFO:teuthology.orchestra.run.smithi033.stdout: mgr/telemetry/nag = false 2023-12-10T18:50:17.068 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-10T18:50:17.068 INFO:teuthology.orchestra.run.smithi033.stdout:[mon] 2023-12-10T18:50:17.068 INFO:teuthology.orchestra.run.smithi033.stdout: debug ms = 1 2023-12-10T18:50:17.068 INFO:teuthology.orchestra.run.smithi033.stdout: debug mon = 20 2023-12-10T18:50:17.068 INFO:teuthology.orchestra.run.smithi033.stdout: debug paxos = 20 2023-12-10T18:50:17.068 INFO:teuthology.orchestra.run.smithi033.stdout: debug auth = 20 2023-12-10T18:50:17.068 INFO:teuthology.orchestra.run.smithi033.stdout: mon data avail warn = 5 2023-12-10T18:50:17.068 INFO:teuthology.orchestra.run.smithi033.stdout: mon mgr mkfs grace = 240 2023-12-10T18:50:17.068 INFO:teuthology.orchestra.run.smithi033.stdout: mon reweight min pgs per osd = 4 2023-12-10T18:50:17.068 INFO:teuthology.orchestra.run.smithi033.stdout: mon osd reporter subtree level = osd 2023-12-10T18:50:17.068 INFO:teuthology.orchestra.run.smithi033.stdout: mon osd prime pg temp = true 2023-12-10T18:50:17.068 INFO:teuthology.orchestra.run.smithi033.stdout: mon reweight min bytes per osd = 10 2023-12-10T18:50:17.068 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-10T18:50:17.068 INFO:teuthology.orchestra.run.smithi033.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-10T18:50:17.069 INFO:teuthology.orchestra.run.smithi033.stdout: auth mon ticket ttl = 660 # 11m 2023-12-10T18:50:17.069 INFO:teuthology.orchestra.run.smithi033.stdout: auth service ticket ttl = 240 # 4m 2023-12-10T18:50:17.069 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-10T18:50:17.069 INFO:teuthology.orchestra.run.smithi033.stdout: # don't complain about insecure global_id in the test suite 2023-12-10T18:50:17.069 INFO:teuthology.orchestra.run.smithi033.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-10T18:50:17.069 INFO:teuthology.orchestra.run.smithi033.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-10T18:50:17.069 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-10T18:50:17.069 INFO:teuthology.orchestra.run.smithi033.stdout: # 1m isn't quite enough 2023-12-10T18:50:17.069 INFO:teuthology.orchestra.run.smithi033.stdout: mon_down_mkfs_grace = 2m 2023-12-10T18:50:17.069 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-10T18:50:17.069 INFO:teuthology.orchestra.run.smithi033.stdout: mon_warn_on_filestore_osds = false 2023-12-10T18:50:17.069 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-10T18:50:17.069 INFO:teuthology.orchestra.run.smithi033.stdout:[client] 2023-12-10T18:50:17.069 INFO:teuthology.orchestra.run.smithi033.stdout: rgw cache enabled = true 2023-12-10T18:50:17.069 INFO:teuthology.orchestra.run.smithi033.stdout: rgw enable ops log = true 2023-12-10T18:50:17.070 INFO:teuthology.orchestra.run.smithi033.stdout: rgw enable usage log = true 2023-12-10T18:50:17.070 INFO:teuthology.orchestra.run.smithi033.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-10T18:50:17.070 INFO:teuthology.orchestra.run.smithi033.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-10T18:50:17.070 INFO:teuthology.orchestra.run.smithi033.stdout: debug rbd = 20 2023-12-10T18:50:17.070 INFO:teuthology.orchestra.run.smithi033.stdout: debug rbd_mirror = 15 2023-12-10T18:50:17.070 INFO:teuthology.orchestra.run.smithi033.stdout: log to stderr = False 2023-12-10T18:50:17.070 INFO:teuthology.orchestra.run.smithi033.stdout: rbd default features = 125 2023-12-10T18:50:17.070 INFO:teuthology.orchestra.run.smithi033.stdout: rbd mirror image policy type = simple 2023-12-10T18:50:17.070 INFO:teuthology.orchestra.run.smithi033.stdout:[mon.a] 2023-12-10T18:50:17.070 INFO:teuthology.orchestra.run.smithi033.stdout:[client.mirror.0] 2023-12-10T18:50:17.070 INFO:teuthology.orchestra.run.smithi033.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-10T18:50:17.070 INFO:teuthology.orchestra.run.smithi033.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-10T18:50:17.070 INFO:teuthology.orchestra.run.smithi033.stdout:[client.mirror.1] 2023-12-10T18:50:17.070 INFO:teuthology.orchestra.run.smithi033.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-10T18:50:17.071 INFO:teuthology.orchestra.run.smithi033.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-10T18:50:17.071 INFO:teuthology.orchestra.run.smithi033.stdout:[client.mirror.2] 2023-12-10T18:50:17.071 INFO:teuthology.orchestra.run.smithi033.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-10T18:50:17.071 INFO:teuthology.orchestra.run.smithi033.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-10T18:50:17.071 INFO:teuthology.orchestra.run.smithi033.stdout:[client.mirror.3] 2023-12-10T18:50:17.071 INFO:teuthology.orchestra.run.smithi033.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-10T18:50:17.071 INFO:teuthology.orchestra.run.smithi033.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-10T18:50:17.071 INFO:teuthology.orchestra.run.smithi076.stdout:[global] 2023-12-10T18:50:17.072 INFO:teuthology.orchestra.run.smithi076.stdout: chdir = "" 2023-12-10T18:50:17.072 INFO:teuthology.orchestra.run.smithi076.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-10T18:50:17.072 INFO:teuthology.orchestra.run.smithi076.stdout: auth supported = cephx 2023-12-10T18:50:17.072 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-10T18:50:17.072 INFO:teuthology.orchestra.run.smithi076.stdout: filestore xattr use omap = true 2023-12-10T18:50:17.072 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-10T18:50:17.072 INFO:teuthology.orchestra.run.smithi076.stdout: mon clock drift allowed = 1.000 2023-12-10T18:50:17.072 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-10T18:50:17.072 INFO:teuthology.orchestra.run.smithi076.stdout: osd crush chooseleaf type = 0 2023-12-10T18:50:17.072 INFO:teuthology.orchestra.run.smithi076.stdout: auth debug = true 2023-12-10T18:50:17.072 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-10T18:50:17.072 INFO:teuthology.orchestra.run.smithi076.stdout: ms die on old message = true 2023-12-10T18:50:17.072 INFO:teuthology.orchestra.run.smithi076.stdout: ms die on bug = true 2023-12-10T18:50:17.072 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-10T18:50:17.072 INFO:teuthology.orchestra.run.smithi076.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-10T18:50:17.073 INFO:teuthology.orchestra.run.smithi076.stdout: mon pg warn max object skew = 0 2023-12-10T18:50:17.073 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-10T18:50:17.073 INFO:teuthology.orchestra.run.smithi076.stdout: # disable pg_autoscaler by default for new pools 2023-12-10T18:50:17.073 INFO:teuthology.orchestra.run.smithi076.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-10T18:50:17.073 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-10T18:50:17.073 INFO:teuthology.orchestra.run.smithi076.stdout: osd pool default size = 2 2023-12-10T18:50:17.073 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-10T18:50:17.073 INFO:teuthology.orchestra.run.smithi076.stdout: mon osd allow primary affinity = true 2023-12-10T18:50:17.073 INFO:teuthology.orchestra.run.smithi076.stdout: mon osd allow pg remap = true 2023-12-10T18:50:17.073 INFO:teuthology.orchestra.run.smithi076.stdout: mon warn on legacy crush tunables = false 2023-12-10T18:50:17.073 INFO:teuthology.orchestra.run.smithi076.stdout: mon warn on crush straw calc version zero = false 2023-12-10T18:50:17.073 INFO:teuthology.orchestra.run.smithi076.stdout: mon warn on no sortbitwise = false 2023-12-10T18:50:17.073 INFO:teuthology.orchestra.run.smithi076.stdout: mon warn on osd down out interval zero = false 2023-12-10T18:50:17.073 INFO:teuthology.orchestra.run.smithi076.stdout: mon warn on too few osds = false 2023-12-10T18:50:17.074 INFO:teuthology.orchestra.run.smithi076.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-10T18:50:17.074 INFO:teuthology.orchestra.run.smithi076.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-10T18:50:17.074 INFO:teuthology.orchestra.run.smithi076.stdout: mon_allow_pool_size_one = true 2023-12-10T18:50:17.074 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-10T18:50:17.074 INFO:teuthology.orchestra.run.smithi076.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-10T18:50:17.074 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-10T18:50:17.074 INFO:teuthology.orchestra.run.smithi076.stdout: osd default data pool replay window = 5 2023-12-10T18:50:17.074 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-10T18:50:17.074 INFO:teuthology.orchestra.run.smithi076.stdout: mon allow pool delete = true 2023-12-10T18:50:17.074 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-10T18:50:17.074 INFO:teuthology.orchestra.run.smithi076.stdout: mon cluster log file level = debug 2023-12-10T18:50:17.074 INFO:teuthology.orchestra.run.smithi076.stdout: debug asserts on shutdown = true 2023-12-10T18:50:17.074 INFO:teuthology.orchestra.run.smithi076.stdout: mon health detail to clog = false 2023-12-10T18:50:17.074 INFO:teuthology.orchestra.run.smithi076.stdout: mon host = 172.21.15.33 2023-12-10T18:50:17.074 INFO:teuthology.orchestra.run.smithi076.stdout: mon client directed command retry = 5 2023-12-10T18:50:17.074 INFO:teuthology.orchestra.run.smithi076.stdout: mon warn on pool no app = False 2023-12-10T18:50:17.075 INFO:teuthology.orchestra.run.smithi076.stdout: ms inject socket failures = 5000 2023-12-10T18:50:17.075 INFO:teuthology.orchestra.run.smithi076.stdout: fsid = d3c37dcf-9605-4f22-911b-14b903774172 2023-12-10T18:50:17.075 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-10T18:50:17.075 INFO:teuthology.orchestra.run.smithi076.stdout:[osd] 2023-12-10T18:50:17.075 INFO:teuthology.orchestra.run.smithi076.stdout: osd journal size = 100 2023-12-10T18:50:17.075 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-10T18:50:17.075 INFO:teuthology.orchestra.run.smithi076.stdout: osd scrub load threshold = 5.0 2023-12-10T18:50:17.075 INFO:teuthology.orchestra.run.smithi076.stdout: osd scrub max interval = 600 2023-12-10T18:50:17.075 INFO:teuthology.orchestra.run.smithi076.stdout: osd mclock profile = high_recovery_ops 2023-12-10T18:50:17.075 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-10T18:50:17.075 INFO:teuthology.orchestra.run.smithi076.stdout: osd recover clone overlap = true 2023-12-10T18:50:17.075 INFO:teuthology.orchestra.run.smithi076.stdout: osd recovery max chunk = 1048576 2023-12-10T18:50:17.075 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-10T18:50:17.075 INFO:teuthology.orchestra.run.smithi076.stdout: osd debug shutdown = true 2023-12-10T18:50:17.075 INFO:teuthology.orchestra.run.smithi076.stdout: osd debug op order = true 2023-12-10T18:50:17.076 INFO:teuthology.orchestra.run.smithi076.stdout: osd debug verify stray on activate = true 2023-12-10T18:50:17.076 INFO:teuthology.orchestra.run.smithi076.stdout: osd debug trim objects = true 2023-12-10T18:50:17.076 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-10T18:50:17.076 INFO:teuthology.orchestra.run.smithi076.stdout: osd open classes on start = true 2023-12-10T18:50:17.076 INFO:teuthology.orchestra.run.smithi076.stdout: osd debug pg log writeout = true 2023-12-10T18:50:17.076 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-10T18:50:17.076 INFO:teuthology.orchestra.run.smithi076.stdout: osd deep scrub update digest min age = 30 2023-12-10T18:50:17.076 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-10T18:50:17.076 INFO:teuthology.orchestra.run.smithi076.stdout: osd map max advance = 10 2023-12-10T18:50:17.076 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-10T18:50:17.076 INFO:teuthology.orchestra.run.smithi076.stdout: journal zero on create = true 2023-12-10T18:50:17.076 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-10T18:50:17.076 INFO:teuthology.orchestra.run.smithi076.stdout: filestore ondisk finisher threads = 3 2023-12-10T18:50:17.076 INFO:teuthology.orchestra.run.smithi076.stdout: filestore apply finisher threads = 3 2023-12-10T18:50:17.076 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-10T18:50:17.076 INFO:teuthology.orchestra.run.smithi076.stdout: bdev debug aio = true 2023-12-10T18:50:17.077 INFO:teuthology.orchestra.run.smithi076.stdout: osd debug misdirected ops = true 2023-12-10T18:50:17.077 INFO:teuthology.orchestra.run.smithi076.stdout: bdev async discard = True 2023-12-10T18:50:17.077 INFO:teuthology.orchestra.run.smithi076.stdout: bdev enable discard = True 2023-12-10T18:50:17.077 INFO:teuthology.orchestra.run.smithi076.stdout: bluestore allocator = stupid 2023-12-10T18:50:17.077 INFO:teuthology.orchestra.run.smithi076.stdout: bluestore block size = 96636764160 2023-12-10T18:50:17.077 INFO:teuthology.orchestra.run.smithi076.stdout: bluestore fsck on mount = True 2023-12-10T18:50:17.077 INFO:teuthology.orchestra.run.smithi076.stdout: debug bluefs = 1/20 2023-12-10T18:50:17.077 INFO:teuthology.orchestra.run.smithi076.stdout: debug bluestore = 1/20 2023-12-10T18:50:17.077 INFO:teuthology.orchestra.run.smithi076.stdout: debug ms = 1 2023-12-10T18:50:17.077 INFO:teuthology.orchestra.run.smithi076.stdout: debug osd = 20 2023-12-10T18:50:17.077 INFO:teuthology.orchestra.run.smithi076.stdout: debug rocksdb = 4/10 2023-12-10T18:50:17.077 INFO:teuthology.orchestra.run.smithi076.stdout: mon osd backfillfull_ratio = 0.85 2023-12-10T18:50:17.077 INFO:teuthology.orchestra.run.smithi076.stdout: mon osd full ratio = 0.9 2023-12-10T18:50:17.077 INFO:teuthology.orchestra.run.smithi076.stdout: mon osd nearfull ratio = 0.8 2023-12-10T18:50:17.077 INFO:teuthology.orchestra.run.smithi076.stdout: osd failsafe full ratio = 0.95 2023-12-10T18:50:17.078 INFO:teuthology.orchestra.run.smithi076.stdout: osd objectstore = bluestore 2023-12-10T18:50:17.078 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-10T18:50:17.078 INFO:teuthology.orchestra.run.smithi076.stdout:[mgr] 2023-12-10T18:50:17.078 INFO:teuthology.orchestra.run.smithi076.stdout: debug ms = 1 2023-12-10T18:50:17.078 INFO:teuthology.orchestra.run.smithi076.stdout: debug mgr = 20 2023-12-10T18:50:17.078 INFO:teuthology.orchestra.run.smithi076.stdout: debug mon = 20 2023-12-10T18:50:17.078 INFO:teuthology.orchestra.run.smithi076.stdout: debug auth = 20 2023-12-10T18:50:17.078 INFO:teuthology.orchestra.run.smithi076.stdout: mon reweight min pgs per osd = 4 2023-12-10T18:50:17.078 INFO:teuthology.orchestra.run.smithi076.stdout: mon reweight min bytes per osd = 10 2023-12-10T18:50:17.078 INFO:teuthology.orchestra.run.smithi076.stdout: mgr/telemetry/nag = false 2023-12-10T18:50:17.078 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-10T18:50:17.078 INFO:teuthology.orchestra.run.smithi076.stdout:[mon] 2023-12-10T18:50:17.078 INFO:teuthology.orchestra.run.smithi076.stdout: debug ms = 1 2023-12-10T18:50:17.078 INFO:teuthology.orchestra.run.smithi076.stdout: debug mon = 20 2023-12-10T18:50:17.078 INFO:teuthology.orchestra.run.smithi076.stdout: debug paxos = 20 2023-12-10T18:50:17.079 INFO:teuthology.orchestra.run.smithi076.stdout: debug auth = 20 2023-12-10T18:50:17.079 INFO:teuthology.orchestra.run.smithi076.stdout: mon data avail warn = 5 2023-12-10T18:50:17.079 INFO:teuthology.orchestra.run.smithi076.stdout: mon mgr mkfs grace = 240 2023-12-10T18:50:17.079 INFO:teuthology.orchestra.run.smithi076.stdout: mon reweight min pgs per osd = 4 2023-12-10T18:50:17.079 INFO:teuthology.orchestra.run.smithi076.stdout: mon osd reporter subtree level = osd 2023-12-10T18:50:17.079 INFO:teuthology.orchestra.run.smithi076.stdout: mon osd prime pg temp = true 2023-12-10T18:50:17.079 INFO:teuthology.orchestra.run.smithi076.stdout: mon reweight min bytes per osd = 10 2023-12-10T18:50:17.079 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-10T18:50:17.079 INFO:teuthology.orchestra.run.smithi076.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-10T18:50:17.079 INFO:teuthology.orchestra.run.smithi076.stdout: auth mon ticket ttl = 660 # 11m 2023-12-10T18:50:17.079 INFO:teuthology.orchestra.run.smithi076.stdout: auth service ticket ttl = 240 # 4m 2023-12-10T18:50:17.079 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-10T18:50:17.079 INFO:teuthology.orchestra.run.smithi076.stdout: # don't complain about insecure global_id in the test suite 2023-12-10T18:50:17.079 INFO:teuthology.orchestra.run.smithi076.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-10T18:50:17.079 INFO:teuthology.orchestra.run.smithi076.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-10T18:50:17.079 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-10T18:50:17.080 INFO:teuthology.orchestra.run.smithi076.stdout: # 1m isn't quite enough 2023-12-10T18:50:17.080 INFO:teuthology.orchestra.run.smithi076.stdout: mon_down_mkfs_grace = 2m 2023-12-10T18:50:17.080 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-10T18:50:17.080 INFO:teuthology.orchestra.run.smithi076.stdout: mon_warn_on_filestore_osds = false 2023-12-10T18:50:17.080 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-10T18:50:17.080 INFO:teuthology.orchestra.run.smithi076.stdout:[client] 2023-12-10T18:50:17.080 INFO:teuthology.orchestra.run.smithi076.stdout: rgw cache enabled = true 2023-12-10T18:50:17.080 INFO:teuthology.orchestra.run.smithi076.stdout: rgw enable ops log = true 2023-12-10T18:50:17.080 INFO:teuthology.orchestra.run.smithi076.stdout: rgw enable usage log = true 2023-12-10T18:50:17.080 INFO:teuthology.orchestra.run.smithi076.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-10T18:50:17.080 INFO:teuthology.orchestra.run.smithi076.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-10T18:50:17.080 INFO:teuthology.orchestra.run.smithi076.stdout: debug rbd = 20 2023-12-10T18:50:17.080 INFO:teuthology.orchestra.run.smithi076.stdout: debug rbd_mirror = 15 2023-12-10T18:50:17.080 INFO:teuthology.orchestra.run.smithi076.stdout: log to stderr = False 2023-12-10T18:50:17.080 INFO:teuthology.orchestra.run.smithi076.stdout: rbd default features = 125 2023-12-10T18:50:17.081 INFO:teuthology.orchestra.run.smithi076.stdout: rbd mirror image policy type = simple 2023-12-10T18:50:17.081 INFO:teuthology.orchestra.run.smithi076.stdout:[mon.a] 2023-12-10T18:50:17.081 INFO:teuthology.orchestra.run.smithi076.stdout:[client.mirror.0] 2023-12-10T18:50:17.081 INFO:teuthology.orchestra.run.smithi076.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-10T18:50:17.081 INFO:teuthology.orchestra.run.smithi076.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-10T18:50:17.081 INFO:teuthology.orchestra.run.smithi076.stdout:[client.mirror.1] 2023-12-10T18:50:17.081 INFO:teuthology.orchestra.run.smithi076.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-10T18:50:17.081 INFO:teuthology.orchestra.run.smithi076.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-10T18:50:17.081 INFO:teuthology.orchestra.run.smithi076.stdout:[client.mirror.2] 2023-12-10T18:50:17.081 INFO:teuthology.orchestra.run.smithi076.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-10T18:50:17.081 INFO:teuthology.orchestra.run.smithi076.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-10T18:50:17.081 INFO:teuthology.orchestra.run.smithi076.stdout:[client.mirror.3] 2023-12-10T18:50:17.081 INFO:teuthology.orchestra.run.smithi076.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-10T18:50:17.081 INFO:teuthology.orchestra.run.smithi076.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-10T18:50:17.083 INFO:tasks.ceph:Creating admin key on cluster1.mon.a... 2023-12-10T18:50:17.083 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster1.keyring 2023-12-10T18:50:17.177 INFO:tasks.ceph:Copying monmap to all nodes... 2023-12-10T18:50:17.177 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-10T18:50:17.177 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/etc/ceph/cluster1.keyring of=/dev/stdout 2023-12-10T18:50:17.183 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-10T18:50:17.183 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/home/ubuntu/cephtest/cluster1.monmap of=/dev/stdout 2023-12-10T18:50:17.231 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi033.front.sepia.ceph.com 2023-12-10T18:50:17.231 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-10T18:50:17.232 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/ceph/cluster1.keyring 2023-12-10T18:50:17.232 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2023-12-10T18:50:17.296 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-10T18:50:17.296 DEBUG:teuthology.orchestra.run.smithi033:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2023-12-10T18:50:17.343 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi076.front.sepia.ceph.com 2023-12-10T18:50:17.343 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2023-12-10T18:50:17.343 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/etc/ceph/cluster1.keyring 2023-12-10T18:50:17.344 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2023-12-10T18:50:17.366 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2023-12-10T18:50:17.366 DEBUG:teuthology.orchestra.run.smithi076:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2023-12-10T18:50:17.415 INFO:tasks.ceph:Setting up mon nodes... 2023-12-10T18:50:17.415 INFO:tasks.ceph:Setting up mgr nodes... 2023-12-10T18:50:17.415 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /var/lib/ceph/mgr/cluster1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster1-x/keyring 2023-12-10T18:50:17.461 INFO:teuthology.orchestra.run.smithi033.stdout:creating /var/lib/ceph/mgr/cluster1-x/keyring 2023-12-10T18:50:17.463 INFO:tasks.ceph:Setting up mds nodes... 2023-12-10T18:50:17.464 INFO:tasks.ceph_client:Setting up client nodes... 2023-12-10T18:50:17.464 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster1.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.0.keyring 2023-12-10T18:50:17.540 INFO:teuthology.orchestra.run.smithi033.stdout:creating /etc/ceph/cluster1.client.0.keyring 2023-12-10T18:50:17.551 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster1.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.keyring 2023-12-10T18:50:17.589 INFO:teuthology.orchestra.run.smithi076.stdout:creating /etc/ceph/cluster1.client.mirror.keyring 2023-12-10T18:50:17.638 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster1.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.0.keyring 2023-12-10T18:50:17.717 INFO:teuthology.orchestra.run.smithi076.stdout:creating /etc/ceph/cluster1.client.mirror.0.keyring 2023-12-10T18:50:17.727 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster1.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.1.keyring 2023-12-10T18:50:17.807 INFO:teuthology.orchestra.run.smithi076.stdout:creating /etc/ceph/cluster1.client.mirror.1.keyring 2023-12-10T18:50:17.819 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster1.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.2.keyring 2023-12-10T18:50:17.893 INFO:teuthology.orchestra.run.smithi076.stdout:creating /etc/ceph/cluster1.client.mirror.2.keyring 2023-12-10T18:50:17.903 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster1.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.3.keyring 2023-12-10T18:50:17.976 INFO:teuthology.orchestra.run.smithi076.stdout:creating /etc/ceph/cluster1.client.mirror.3.keyring 2023-12-10T18:50:17.986 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster1.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.4.keyring 2023-12-10T18:50:18.066 INFO:teuthology.orchestra.run.smithi076.stdout:creating /etc/ceph/cluster1.client.mirror.4.keyring 2023-12-10T18:50:18.075 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster1.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.5.keyring 2023-12-10T18:50:18.149 INFO:teuthology.orchestra.run.smithi076.stdout:creating /etc/ceph/cluster1.client.mirror.5.keyring 2023-12-10T18:50:18.160 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster1.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.6.keyring 2023-12-10T18:50:18.234 INFO:teuthology.orchestra.run.smithi076.stdout:creating /etc/ceph/cluster1.client.mirror.6.keyring 2023-12-10T18:50:18.245 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-12-10T18:50:18.245 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi033.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2023-12-10T18:50:18.245 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /var/lib/ceph/osd/cluster1-0 2023-12-10T18:50:18.260 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-10T18:50:18.260 INFO:tasks.ceph:role: cluster1.osd.0 2023-12-10T18:50:18.260 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi033.front.sepia.ceph.com 2023-12-10T18:50:18.260 DEBUG:teuthology.orchestra.run.smithi033:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-12-10T18:50:18.323 INFO:teuthology.orchestra.run.smithi033.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-12-10T18:50:18.323 INFO:teuthology.orchestra.run.smithi033.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-10T18:50:18.323 INFO:teuthology.orchestra.run.smithi033.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-10T18:50:18.323 INFO:teuthology.orchestra.run.smithi033.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-12-10T18:50:18.323 INFO:teuthology.orchestra.run.smithi033.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-10T18:50:18.323 INFO:teuthology.orchestra.run.smithi033.stdout: = sunit=0 swidth=0 blks 2023-12-10T18:50:18.323 INFO:teuthology.orchestra.run.smithi033.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-10T18:50:18.323 INFO:teuthology.orchestra.run.smithi033.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-10T18:50:18.323 INFO:teuthology.orchestra.run.smithi033.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-10T18:50:18.323 INFO:teuthology.orchestra.run.smithi033.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-10T18:50:18.803 INFO:teuthology.orchestra.run.smithi033.stdout:Discarding blocks...Done. 2023-12-10T18:50:18.805 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi033.front.sepia.ceph.com -o noatime 2023-12-10T18:50:18.805 DEBUG:teuthology.orchestra.run.smithi033:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster1-0 2023-12-10T18:50:18.945 DEBUG:teuthology.orchestra.run.smithi033:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-0 2023-12-10T18:50:18.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-10T18:50:18.958 INFO:teuthology.orchestra.run.smithi033.stderr:sudo: /sbin/restorecon: command not found 2023-12-10T18:50:18.958 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /var/lib/ceph/osd/cluster1-1 2023-12-10T18:50:19.015 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-10T18:50:19.015 INFO:tasks.ceph:role: cluster1.osd.1 2023-12-10T18:50:19.015 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi033.front.sepia.ceph.com 2023-12-10T18:50:19.015 DEBUG:teuthology.orchestra.run.smithi033:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-12-10T18:50:19.079 INFO:teuthology.orchestra.run.smithi033.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-12-10T18:50:19.080 INFO:teuthology.orchestra.run.smithi033.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-10T18:50:19.080 INFO:teuthology.orchestra.run.smithi033.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-10T18:50:19.080 INFO:teuthology.orchestra.run.smithi033.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-12-10T18:50:19.080 INFO:teuthology.orchestra.run.smithi033.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-10T18:50:19.080 INFO:teuthology.orchestra.run.smithi033.stdout: = sunit=0 swidth=0 blks 2023-12-10T18:50:19.080 INFO:teuthology.orchestra.run.smithi033.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-10T18:50:19.080 INFO:teuthology.orchestra.run.smithi033.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-10T18:50:19.080 INFO:teuthology.orchestra.run.smithi033.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-10T18:50:19.080 INFO:teuthology.orchestra.run.smithi033.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-10T18:50:19.526 INFO:teuthology.orchestra.run.smithi033.stdout:Discarding blocks...Done. 2023-12-10T18:50:19.528 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi033.front.sepia.ceph.com -o noatime 2023-12-10T18:50:19.528 DEBUG:teuthology.orchestra.run.smithi033:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster1-1 2023-12-10T18:50:19.623 DEBUG:teuthology.orchestra.run.smithi033:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-1 2023-12-10T18:50:19.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-10T18:50:19.635 INFO:teuthology.orchestra.run.smithi033.stderr:sudo: /sbin/restorecon: command not found 2023-12-10T18:50:19.636 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /var/lib/ceph/osd/cluster1-2 2023-12-10T18:50:19.691 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-10T18:50:19.691 INFO:tasks.ceph:role: cluster1.osd.2 2023-12-10T18:50:19.691 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi033.front.sepia.ceph.com 2023-12-10T18:50:19.691 DEBUG:teuthology.orchestra.run.smithi033:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-12-10T18:50:19.754 INFO:teuthology.orchestra.run.smithi033.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-12-10T18:50:19.754 INFO:teuthology.orchestra.run.smithi033.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-10T18:50:19.754 INFO:teuthology.orchestra.run.smithi033.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-10T18:50:19.754 INFO:teuthology.orchestra.run.smithi033.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-12-10T18:50:19.754 INFO:teuthology.orchestra.run.smithi033.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-10T18:50:19.754 INFO:teuthology.orchestra.run.smithi033.stdout: = sunit=0 swidth=0 blks 2023-12-10T18:50:19.754 INFO:teuthology.orchestra.run.smithi033.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-10T18:50:19.754 INFO:teuthology.orchestra.run.smithi033.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-10T18:50:19.754 INFO:teuthology.orchestra.run.smithi033.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-10T18:50:19.754 INFO:teuthology.orchestra.run.smithi033.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-10T18:50:20.219 INFO:teuthology.orchestra.run.smithi033.stdout:Discarding blocks...Done. 2023-12-10T18:50:20.222 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi033.front.sepia.ceph.com -o noatime 2023-12-10T18:50:20.222 DEBUG:teuthology.orchestra.run.smithi033:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster1-2 2023-12-10T18:50:20.348 DEBUG:teuthology.orchestra.run.smithi033:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-2 2023-12-10T18:50:20.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-10T18:50:20.361 INFO:teuthology.orchestra.run.smithi033.stderr:sudo: /sbin/restorecon: command not found 2023-12-10T18:50:20.361 DEBUG:teuthology.orchestra.run.smithi033:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster1.monmap 2023-12-10T18:50:20.481 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-10T18:50:20.476+0000 7f09f3cc7640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-0/keyring: can't open /var/lib/ceph/osd/cluster1-0/keyring: (2) No such file or directory 2023-12-10T18:50:20.483 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-10T18:50:20.480+0000 7f09f3cc7640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-0/keyring 2023-12-10T18:50:20.483 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-10T18:50:20.480+0000 7f09f3cc7640 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2023-12-10T18:50:20.483 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-10T18:50:20.480+0000 7f09f3cc7640 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2023-12-10T18:50:20.483 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-10T18:50:20.480+0000 7f09f3cc7640 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2023-12-10T18:50:20.491 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-10T18:50:20.488+0000 7f09f3cc7640 -1 bluestore(/var/lib/ceph/osd/cluster1-0) _read_fsid unparsable uuid 2023-12-10T18:50:23.533 DEBUG:teuthology.orchestra.run.smithi033:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-0 2023-12-10T18:50:23.548 DEBUG:teuthology.orchestra.run.smithi033:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster1.monmap 2023-12-10T18:50:23.632 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-10T18:50:23.628+0000 7fa954aeb640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-1/keyring: can't open /var/lib/ceph/osd/cluster1-1/keyring: (2) No such file or directory 2023-12-10T18:50:23.633 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-10T18:50:23.628+0000 7fa954aeb640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-1/keyring 2023-12-10T18:50:23.633 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-10T18:50:23.628+0000 7fa954aeb640 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2023-12-10T18:50:23.633 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-10T18:50:23.628+0000 7fa954aeb640 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2023-12-10T18:50:23.633 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-10T18:50:23.628+0000 7fa954aeb640 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2023-12-10T18:50:23.634 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-10T18:50:23.628+0000 7fa954aeb640 -1 bluestore(/var/lib/ceph/osd/cluster1-1) _read_fsid unparsable uuid 2023-12-10T18:50:26.674 DEBUG:teuthology.orchestra.run.smithi033:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-1 2023-12-10T18:50:26.732 DEBUG:teuthology.orchestra.run.smithi033:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster1.monmap 2023-12-10T18:50:26.814 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-10T18:50:26.812+0000 7faa0967f640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-2/keyring: can't open /var/lib/ceph/osd/cluster1-2/keyring: (2) No such file or directory 2023-12-10T18:50:26.815 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-10T18:50:26.812+0000 7faa0967f640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-2/keyring 2023-12-10T18:50:26.815 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-10T18:50:26.812+0000 7faa0967f640 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2023-12-10T18:50:26.815 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-10T18:50:26.812+0000 7faa0967f640 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2023-12-10T18:50:26.815 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-10T18:50:26.812+0000 7faa0967f640 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2023-12-10T18:50:26.816 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-10T18:50:26.812+0000 7faa0967f640 -1 bluestore(/var/lib/ceph/osd/cluster1-2) _read_fsid unparsable uuid 2023-12-10T18:50:29.824 DEBUG:teuthology.orchestra.run.smithi033:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-2 2023-12-10T18:50:29.884 INFO:tasks.ceph:Reading keys from all nodes... 2023-12-10T18:50:29.884 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-10T18:50:29.884 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/mgr/cluster1-x/keyring of=/dev/stdout 2023-12-10T18:50:29.939 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-10T18:50:29.939 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/osd/cluster1-0/keyring of=/dev/stdout 2023-12-10T18:50:29.994 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-10T18:50:29.994 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/osd/cluster1-1/keyring of=/dev/stdout 2023-12-10T18:50:30.054 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-10T18:50:30.054 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/osd/cluster1-2/keyring of=/dev/stdout 2023-12-10T18:50:30.110 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-10T18:50:30.111 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/etc/ceph/cluster1.client.0.keyring of=/dev/stdout 2023-12-10T18:50:30.159 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2023-12-10T18:50:30.159 DEBUG:teuthology.orchestra.run.smithi076:> dd if=/etc/ceph/cluster1.client.mirror.keyring of=/dev/stdout 2023-12-10T18:50:30.165 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2023-12-10T18:50:30.166 DEBUG:teuthology.orchestra.run.smithi076:> dd if=/etc/ceph/cluster1.client.mirror.0.keyring of=/dev/stdout 2023-12-10T18:50:30.214 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2023-12-10T18:50:30.214 DEBUG:teuthology.orchestra.run.smithi076:> dd if=/etc/ceph/cluster1.client.mirror.1.keyring of=/dev/stdout 2023-12-10T18:50:30.262 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2023-12-10T18:50:30.262 DEBUG:teuthology.orchestra.run.smithi076:> dd if=/etc/ceph/cluster1.client.mirror.2.keyring of=/dev/stdout 2023-12-10T18:50:30.310 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2023-12-10T18:50:30.311 DEBUG:teuthology.orchestra.run.smithi076:> dd if=/etc/ceph/cluster1.client.mirror.3.keyring of=/dev/stdout 2023-12-10T18:50:30.358 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2023-12-10T18:50:30.358 DEBUG:teuthology.orchestra.run.smithi076:> dd if=/etc/ceph/cluster1.client.mirror.4.keyring of=/dev/stdout 2023-12-10T18:50:30.407 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2023-12-10T18:50:30.407 DEBUG:teuthology.orchestra.run.smithi076:> dd if=/etc/ceph/cluster1.client.mirror.5.keyring of=/dev/stdout 2023-12-10T18:50:30.454 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2023-12-10T18:50:30.454 DEBUG:teuthology.orchestra.run.smithi076:> dd if=/etc/ceph/cluster1.client.mirror.6.keyring of=/dev/stdout 2023-12-10T18:50:30.502 INFO:tasks.ceph:Adding keys to all mons... 2023-12-10T18:50:30.503 DEBUG:teuthology.orchestra.run.smithi033:> sudo tee -a /etc/ceph/cluster1.keyring 2023-12-10T18:50:30.515 INFO:teuthology.orchestra.run.smithi033.stdout:[mgr.x] 2023-12-10T18:50:30.515 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQBpCHZlBiKDGxAAzPiYrbgMXGm540VymlVGNA== 2023-12-10T18:50:30.515 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.0] 2023-12-10T18:50:30.515 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQBsCHZl58+4HBAAjJ2bIVcJGoTZRSisb8RVnA== 2023-12-10T18:50:30.515 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.1] 2023-12-10T18:50:30.515 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQBvCHZlxmjBJRAAxO1cO3ZOycuSjd3miYGWYg== 2023-12-10T18:50:30.515 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.2] 2023-12-10T18:50:30.515 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQByCHZlbqmbMBAAjknB0m0SYoGXQkRyoK1Aiw== 2023-12-10T18:50:30.515 INFO:teuthology.orchestra.run.smithi033.stdout:[client.0] 2023-12-10T18:50:30.515 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQBpCHZlHcdBIBAAAEDtVSymg6r/8f1HR79I7w== 2023-12-10T18:50:30.515 INFO:teuthology.orchestra.run.smithi033.stdout:[client.mirror] 2023-12-10T18:50:30.516 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQBpCHZlukn5IhAAVam+mhPU05gN36oBgaBbag== 2023-12-10T18:50:30.516 INFO:teuthology.orchestra.run.smithi033.stdout:[client.mirror.0] 2023-12-10T18:50:30.516 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQBpCHZlyyqVKhAAif2T0ChYfzY/KMnp9Q35Sg== 2023-12-10T18:50:30.516 INFO:teuthology.orchestra.run.smithi033.stdout:[client.mirror.1] 2023-12-10T18:50:30.516 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQBpCHZlzXz6LxAAVd7Uh4mATM05DtSjlYthJw== 2023-12-10T18:50:30.516 INFO:teuthology.orchestra.run.smithi033.stdout:[client.mirror.2] 2023-12-10T18:50:30.516 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQBpCHZlNuQbNRAA/dGJQv/ryls/ICFExzk+Gg== 2023-12-10T18:50:30.516 INFO:teuthology.orchestra.run.smithi033.stdout:[client.mirror.3] 2023-12-10T18:50:30.516 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQBpCHZlt+IGOhAAJ5QPB/ViM7rmYVPcEM374w== 2023-12-10T18:50:30.516 INFO:teuthology.orchestra.run.smithi033.stdout:[client.mirror.4] 2023-12-10T18:50:30.516 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQBqCHZlw1eoAxAAsEa+vimsphaVpZ9GaYxlGA== 2023-12-10T18:50:30.516 INFO:teuthology.orchestra.run.smithi033.stdout:[client.mirror.5] 2023-12-10T18:50:30.516 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQBqCHZlM4nECBAAQ3EpbM4FT72vXV0zRItkXQ== 2023-12-10T18:50:30.516 INFO:teuthology.orchestra.run.smithi033.stdout:[client.mirror.6] 2023-12-10T18:50:30.516 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQBqCHZlB4vPDRAA3RyAPuOsDNhqLqJHXOYpfA== 2023-12-10T18:50:30.517 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-10T18:50:30.605 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-10T18:50:30.693 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-10T18:50:30.777 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-10T18:50:30.861 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-10T18:50:30.951 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-10T18:50:30.994 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-10T18:50:31.082 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-10T18:50:31.175 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-10T18:50:31.222 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-10T18:50:31.314 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-10T18:50:31.357 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-10T18:50:31.440 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-10T18:50:31.523 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-12-10T18:50:31.524 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /var/lib/ceph/mon/cluster1-a 2023-12-10T18:50:31.576 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster1 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster1.monmap --keyring /etc/ceph/cluster1.keyring 2023-12-10T18:50:31.667 DEBUG:teuthology.orchestra.run.smithi033:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster1-a 2023-12-10T18:50:31.678 DEBUG:teuthology.orchestra.run.smithi033:> rm -- /home/ubuntu/cephtest/cluster1.monmap 2023-12-10T18:50:31.731 INFO:tasks.ceph:Starting mon daemons in cluster cluster1... 2023-12-10T18:50:31.731 INFO:tasks.ceph.cluster1.mon.a:Restarting daemon 2023-12-10T18:50:31.731 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster1 -i a 2023-12-10T18:50:31.775 INFO:tasks.ceph.cluster1.mon.a:Started 2023-12-10T18:50:31.776 INFO:tasks.ceph:Starting mgr daemons in cluster cluster1... 2023-12-10T18:50:31.776 INFO:tasks.ceph.cluster1.mgr.x:Restarting daemon 2023-12-10T18:50:31.776 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster1 -i x 2023-12-10T18:50:31.778 INFO:tasks.ceph.cluster1.mgr.x:Started 2023-12-10T18:50:31.778 INFO:tasks.ceph:Setting crush tunables to default 2023-12-10T18:50:31.778 DEBUG:teuthology.orchestra.run.smithi033:> sudo ceph --cluster cluster1 osd crush tunables default 2023-12-10T18:50:32.036 INFO:teuthology.orchestra.run.smithi033.stderr:adjusted tunables profile to default 2023-12-10T18:50:32.055 INFO:tasks.ceph:check_enable_crimson: False 2023-12-10T18:50:32.055 INFO:tasks.ceph:Starting osd daemons in cluster cluster1... 2023-12-10T18:50:32.055 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-10T18:50:32.055 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/osd/cluster1-0/fsid of=/dev/stdout 2023-12-10T18:50:32.065 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-10T18:50:32.065 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/osd/cluster1-1/fsid of=/dev/stdout 2023-12-10T18:50:32.121 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-10T18:50:32.121 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/osd/cluster1-2/fsid of=/dev/stdout 2023-12-10T18:50:32.176 DEBUG:teuthology.orchestra.run.smithi033:> sudo ceph --cluster cluster1 osd new cade80c6-664a-4a0a-8298-0c400418aad4 0 2023-12-10T18:50:32.191 INFO:tasks.ceph.cluster1.mgr.x.smithi033.stderr:2023-12-10T18:50:32.188+0000 7f6b0cebe180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-10T18:50:32.392 INFO:teuthology.orchestra.run.smithi033.stdout:0 2023-12-10T18:50:32.410 DEBUG:teuthology.orchestra.run.smithi033:> sudo ceph --cluster cluster1 osd new 90ff3f91-f341-41a0-82fb-67b8ffe22dc5 1 2023-12-10T18:50:32.611 INFO:teuthology.orchestra.run.smithi033.stdout:1 2023-12-10T18:50:32.630 DEBUG:teuthology.orchestra.run.smithi033:> sudo ceph --cluster cluster1 osd new 50c019c1-d5ef-4bd8-9acf-f6ae6b3f538a 2 2023-12-10T18:50:32.788 INFO:tasks.ceph.cluster1.mgr.x.smithi033.stderr:2023-12-10T18:50:32.784+0000 7f6b0cebe180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-10T18:50:32.831 INFO:teuthology.orchestra.run.smithi033.stdout:2 2023-12-10T18:50:32.850 INFO:tasks.ceph.cluster1.osd.0:Restarting daemon 2023-12-10T18:50:32.850 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 0 2023-12-10T18:50:32.852 INFO:tasks.ceph.cluster1.osd.0:Started 2023-12-10T18:50:32.852 INFO:tasks.ceph.cluster1.osd.1:Restarting daemon 2023-12-10T18:50:32.852 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 1 2023-12-10T18:50:32.854 INFO:tasks.ceph.cluster1.osd.1:Started 2023-12-10T18:50:32.854 INFO:tasks.ceph.cluster1.osd.2:Restarting daemon 2023-12-10T18:50:32.855 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 2 2023-12-10T18:50:32.856 INFO:tasks.ceph.cluster1.osd.2:Started 2023-12-10T18:50:32.857 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2023-12-10T18:50:33.003 INFO:tasks.ceph.cluster1.mgr.x.smithi033.stderr:2023-12-10T18:50:33.000+0000 7f6b0cebe180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-10T18:50:33.232 INFO:tasks.ceph.cluster1.osd.1.smithi033.stderr:2023-12-10T18:50:33.228+0000 7f06b6afe640 -1 Falling back to public interface 2023-12-10T18:50:33.257 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-10T18:50:33.258 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":5,"fsid":"d3c37dcf-9605-4f22-911b-14b903774172","created":"2023-12-10T18:50:31.918913+0000","modified":"2023-12-10T18:50:32.830957+0000","last_up_change":"0.000000","last_in_change":"2023-12-10T18:50:32.830957+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cade80c6-664a-4a0a-8298-0c400418aad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"90ff3f91-f341-41a0-82fb-67b8ffe22dc5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"50c019c1-d5ef-4bd8-9acf-f6ae6b3f538a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-10T18:50:33.271 INFO:tasks.ceph.ceph_manager.cluster1:[] 2023-12-10T18:50:33.271 INFO:tasks.ceph:Waiting for OSDs to come up 2023-12-10T18:50:33.468 INFO:tasks.ceph.cluster1.osd.0.smithi033.stderr:2023-12-10T18:50:33.464+0000 7f8380f27640 -1 Falling back to public interface 2023-12-10T18:50:33.491 INFO:tasks.ceph.cluster1.osd.2.smithi033.stderr:2023-12-10T18:50:33.488+0000 7fef9c174640 -1 Falling back to public interface 2023-12-10T18:50:33.573 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2023-12-10T18:50:33.601 INFO:tasks.ceph.cluster1.mgr.x.smithi033.stderr:2023-12-10T18:50:33.596+0000 7f6b0cebe180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-10T18:50:33.658 INFO:tasks.ceph.cluster1.mgr.x.smithi033.stderr:2023-12-10T18:50:33.652+0000 7f6b0cebe180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-10T18:50:33.717 INFO:tasks.ceph.cluster1.mgr.x.smithi033.stderr:2023-12-10T18:50:33.712+0000 7f6b0cebe180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-10T18:50:33.751 INFO:teuthology.misc.health.smithi033.stdout: 2023-12-10T18:50:33.751 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":5,"fsid":"d3c37dcf-9605-4f22-911b-14b903774172","created":"2023-12-10T18:50:31.918913+0000","modified":"2023-12-10T18:50:32.830957+0000","last_up_change":"0.000000","last_in_change":"2023-12-10T18:50:32.830957+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cade80c6-664a-4a0a-8298-0c400418aad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"90ff3f91-f341-41a0-82fb-67b8ffe22dc5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"50c019c1-d5ef-4bd8-9acf-f6ae6b3f538a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-10T18:50:33.766 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-12-10T18:50:33.799 INFO:tasks.ceph.cluster1.mgr.x.smithi033.stderr:2023-12-10T18:50:33.796+0000 7f6b0cebe180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-10T18:50:33.853 INFO:tasks.ceph.cluster1.mgr.x.smithi033.stderr:2023-12-10T18:50:33.848+0000 7f6b0cebe180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-10T18:50:33.934 INFO:tasks.ceph.cluster1.mgr.x.smithi033.stderr:2023-12-10T18:50:33.928+0000 7f6b0cebe180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-10T18:50:33.989 INFO:tasks.ceph.cluster1.mgr.x.smithi033.stderr:2023-12-10T18:50:33.984+0000 7f6b0cebe180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-10T18:50:34.134 INFO:tasks.ceph.cluster1.mgr.x.smithi033.stderr:2023-12-10T18:50:34.128+0000 7f6b0cebe180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-10T18:50:34.196 INFO:tasks.ceph.cluster1.mgr.x.smithi033.stderr:2023-12-10T18:50:34.192+0000 7f6b0cebe180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-10T18:50:34.799 INFO:tasks.ceph.cluster1.mgr.x.smithi033.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-12-10T18:50:34.799 INFO:tasks.ceph.cluster1.mgr.x.smithi033.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-12-10T18:50:34.799 INFO:tasks.ceph.cluster1.mgr.x.smithi033.stderr: from numpy import show_config as show_numpy_config 2023-12-10T18:50:34.894 INFO:tasks.ceph.cluster1.mgr.x.smithi033.stderr:2023-12-10T18:50:34.892+0000 7f6b0cebe180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-12-10T18:50:35.069 INFO:tasks.ceph.cluster1.mgr.x.smithi033.stderr:2023-12-10T18:50:35.064+0000 7f6b0cebe180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-10T18:50:35.242 INFO:tasks.ceph.cluster1.mgr.x.smithi033.stderr:2023-12-10T18:50:35.236+0000 7f6b0cebe180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-10T18:50:35.313 INFO:tasks.ceph.cluster1.mgr.x.smithi033.stderr:2023-12-10T18:50:35.308+0000 7f6b0cebe180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-10T18:50:35.452 INFO:tasks.ceph.cluster1.mgr.x.smithi033.stderr:2023-12-10T18:50:35.448+0000 7f6b0cebe180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-10T18:50:36.092 INFO:tasks.ceph.cluster1.mgr.x.smithi033.stderr:2023-12-10T18:50:36.088+0000 7f6b0cebe180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-10T18:50:36.170 INFO:tasks.ceph.cluster1.mgr.x.smithi033.stderr:2023-12-10T18:50:36.168+0000 7f6b0cebe180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-10T18:50:36.353 INFO:tasks.ceph.cluster1.mgr.x.smithi033.stderr:2023-12-10T18:50:36.348+0000 7f6b0cebe180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-10T18:50:36.438 INFO:tasks.ceph.cluster1.mgr.x.smithi033.stderr:2023-12-10T18:50:36.432+0000 7f6b0cebe180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-10T18:50:36.758 INFO:tasks.ceph.cluster1.mgr.x.smithi033.stderr:2023-12-10T18:50:36.756+0000 7f6b0cebe180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-10T18:50:36.823 INFO:tasks.ceph.cluster1.mgr.x.smithi033.stderr:2023-12-10T18:50:36.820+0000 7f6b0cebe180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-10T18:50:37.157 INFO:tasks.ceph.cluster1.osd.1.smithi033.stderr:2023-12-10T18:50:37.152+0000 7f06b6afe640 -1 osd.1 0 log_to_monitors true 2023-12-10T18:50:37.377 INFO:tasks.ceph.cluster1.osd.0.smithi033.stderr:2023-12-10T18:50:37.372+0000 7f8380f27640 -1 osd.0 0 log_to_monitors true 2023-12-10T18:50:37.377 INFO:tasks.ceph.cluster1.osd.2.smithi033.stderr:2023-12-10T18:50:37.372+0000 7fef9c174640 -1 osd.2 0 log_to_monitors true 2023-12-10T18:50:37.867 INFO:tasks.ceph.cluster1.mgr.x.smithi033.stderr:2023-12-10T18:50:37.864+0000 7f6ab4609640 -1 mgr.server handle_report got status from non-daemon mon.a 2023-12-10T18:50:40.069 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2023-12-10T18:50:40.527 INFO:teuthology.misc.health.smithi033.stdout: 2023-12-10T18:50:40.527 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":7,"fsid":"d3c37dcf-9605-4f22-911b-14b903774172","created":"2023-12-10T18:50:31.918913+0000","modified":"2023-12-10T18:50:38.905164+0000","last_up_change":"0.000000","last_in_change":"2023-12-10T18:50:32.830957+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cade80c6-664a-4a0a-8298-0c400418aad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"90ff3f91-f341-41a0-82fb-67b8ffe22dc5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"50c019c1-d5ef-4bd8-9acf-f6ae6b3f538a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-10T18:50:40.554 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-12-10T18:50:40.982 INFO:tasks.ceph.cluster1.osd.0.smithi033.stderr:2023-12-10T18:50:40.976+0000 7f837dee3640 -1 osd.0 0 waiting for initial osdmap 2023-12-10T18:50:41.007 INFO:tasks.ceph.cluster1.osd.0.smithi033.stderr:2023-12-10T18:50:41.004+0000 7f8374cfa640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-10T18:50:41.013 INFO:tasks.ceph.cluster1.osd.2.smithi033.stderr:2023-12-10T18:50:41.008+0000 7fef99130640 -1 osd.2 0 waiting for initial osdmap 2023-12-10T18:50:41.019 INFO:tasks.ceph.cluster1.osd.1.smithi033.stderr:2023-12-10T18:50:41.016+0000 7f06b2284640 -1 osd.1 0 waiting for initial osdmap 2023-12-10T18:50:41.055 INFO:tasks.ceph.cluster1.osd.2.smithi033.stderr:2023-12-10T18:50:41.052+0000 7fef8ff47640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-10T18:50:41.055 INFO:tasks.ceph.cluster1.osd.1.smithi033.stderr:2023-12-10T18:50:41.052+0000 7f06aa8d1640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-10T18:50:46.857 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2023-12-10T18:50:47.154 INFO:teuthology.misc.health.smithi033.stdout: 2023-12-10T18:50:47.154 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":11,"fsid":"d3c37dcf-9605-4f22-911b-14b903774172","created":"2023-12-10T18:50:31.918913+0000","modified":"2023-12-10T18:50:45.040841+0000","last_up_change":"2023-12-10T18:50:42.021651+0000","last_in_change":"2023-12-10T18:50:32.830957+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-10T18:50:42.925167+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cade80c6-664a-4a0a-8298-0c400418aad4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":223693307},{"type":"v1","addr":"172.21.15.33:6809","nonce":223693307}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":223693307},{"type":"v1","addr":"172.21.15.33:6811","nonce":223693307}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":223693307},{"type":"v1","addr":"172.21.15.33:6815","nonce":223693307}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":223693307},{"type":"v1","addr":"172.21.15.33:6813","nonce":223693307}]},"public_addr":"172.21.15.33:6809/223693307","cluster_addr":"172.21.15.33:6811/223693307","heartbeat_back_addr":"172.21.15.33:6815/223693307","heartbeat_front_addr":"172.21.15.33:6813/223693307","state":["exists","up"]},{"osd":1,"uuid":"90ff3f91-f341-41a0-82fb-67b8ffe22dc5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":3180911142},{"type":"v1","addr":"172.21.15.33:6801","nonce":3180911142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":3180911142},{"type":"v1","addr":"172.21.15.33:6803","nonce":3180911142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":3180911142},{"type":"v1","addr":"172.21.15.33:6807","nonce":3180911142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":3180911142},{"type":"v1","addr":"172.21.15.33:6805","nonce":3180911142}]},"public_addr":"172.21.15.33:6801/3180911142","cluster_addr":"172.21.15.33:6803/3180911142","heartbeat_back_addr":"172.21.15.33:6807/3180911142","heartbeat_front_addr":"172.21.15.33:6805/3180911142","state":["exists","up"]},{"osd":2,"uuid":"50c019c1-d5ef-4bd8-9acf-f6ae6b3f538a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":1946811008},{"type":"v1","addr":"172.21.15.33:6817","nonce":1946811008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":1946811008},{"type":"v1","addr":"172.21.15.33:6819","nonce":1946811008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":1946811008},{"type":"v1","addr":"172.21.15.33:6823","nonce":1946811008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":1946811008},{"type":"v1","addr":"172.21.15.33:6821","nonce":1946811008}]},"public_addr":"172.21.15.33:6817/1946811008","cluster_addr":"172.21.15.33:6819/1946811008","heartbeat_back_addr":"172.21.15.33:6823/1946811008","heartbeat_front_addr":"172.21.15.33:6821/1946811008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-10T18:50:38.334630+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-10T18:50:38.120123+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-10T18:50:38.338202+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-10T18:50:47.171 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-12-10T18:50:47.171 INFO:tasks.ceph:Creating RBD pool 2023-12-10T18:50:47.172 DEBUG:teuthology.orchestra.run.smithi033:> sudo ceph --cluster cluster1 osd pool create rbd 8 2023-12-10T18:50:47.869 INFO:teuthology.orchestra.run.smithi033.stderr:pool 'rbd' created 2023-12-10T18:50:47.898 DEBUG:teuthology.orchestra.run.smithi033:> sudo ceph --cluster cluster1 osd pool application enable rbd rbd --yes-i-really-mean-it 2023-12-10T18:50:48.876 INFO:teuthology.orchestra.run.smithi033.stderr:enabled application 'rbd' on pool 'rbd' 2023-12-10T18:50:48.908 INFO:tasks.ceph:Starting mds daemons in cluster cluster1... 2023-12-10T18:50:48.908 INFO:tasks.ceph:Waiting until cluster1 daemons up and pgs clean... 2023-12-10T18:50:48.908 INFO:tasks.ceph.ceph_manager.cluster1:waiting for mgr available 2023-12-10T18:50:48.909 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 mgr dump --format=json 2023-12-10T18:50:48.909 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-12-10T18:50:49.245 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-10T18:50:49.260 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":5,"active_gid":4099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":1869648584},{"type":"v1","addr":"172.21.15.33:6825","nonce":1869648584}]},"active_addr":"172.21.15.33:6825/1869648584","active_change":"2023-12-10T18:50:36.844546+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.33:0","nonce":3079902249}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.33:0","nonce":3653886748}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.33:0","nonce":391724205}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.33:0","nonce":2418902936}]}]} 2023-12-10T18:50:49.263 INFO:tasks.ceph.ceph_manager.cluster1:mgr available! 2023-12-10T18:50:49.263 INFO:tasks.ceph.ceph_manager.cluster1:waiting for all up 2023-12-10T18:50:49.263 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2023-12-10T18:50:49.556 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-10T18:50:49.557 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":13,"fsid":"d3c37dcf-9605-4f22-911b-14b903774172","created":"2023-12-10T18:50:31.918913+0000","modified":"2023-12-10T18:50:48.869027+0000","last_up_change":"2023-12-10T18:50:42.021651+0000","last_in_change":"2023-12-10T18:50:32.830957+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-10T18:50:42.925167+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-10T18:50:47.511893+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cade80c6-664a-4a0a-8298-0c400418aad4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":223693307},{"type":"v1","addr":"172.21.15.33:6809","nonce":223693307}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":223693307},{"type":"v1","addr":"172.21.15.33:6811","nonce":223693307}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":223693307},{"type":"v1","addr":"172.21.15.33:6815","nonce":223693307}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":223693307},{"type":"v1","addr":"172.21.15.33:6813","nonce":223693307}]},"public_addr":"172.21.15.33:6809/223693307","cluster_addr":"172.21.15.33:6811/223693307","heartbeat_back_addr":"172.21.15.33:6815/223693307","heartbeat_front_addr":"172.21.15.33:6813/223693307","state":["exists","up"]},{"osd":1,"uuid":"90ff3f91-f341-41a0-82fb-67b8ffe22dc5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":3180911142},{"type":"v1","addr":"172.21.15.33:6801","nonce":3180911142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":3180911142},{"type":"v1","addr":"172.21.15.33:6803","nonce":3180911142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":3180911142},{"type":"v1","addr":"172.21.15.33:6807","nonce":3180911142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":3180911142},{"type":"v1","addr":"172.21.15.33:6805","nonce":3180911142}]},"public_addr":"172.21.15.33:6801/3180911142","cluster_addr":"172.21.15.33:6803/3180911142","heartbeat_back_addr":"172.21.15.33:6807/3180911142","heartbeat_front_addr":"172.21.15.33:6805/3180911142","state":["exists","up"]},{"osd":2,"uuid":"50c019c1-d5ef-4bd8-9acf-f6ae6b3f538a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":1946811008},{"type":"v1","addr":"172.21.15.33:6817","nonce":1946811008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":1946811008},{"type":"v1","addr":"172.21.15.33:6819","nonce":1946811008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":1946811008},{"type":"v1","addr":"172.21.15.33:6823","nonce":1946811008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":1946811008},{"type":"v1","addr":"172.21.15.33:6821","nonce":1946811008}]},"public_addr":"172.21.15.33:6817/1946811008","cluster_addr":"172.21.15.33:6819/1946811008","heartbeat_back_addr":"172.21.15.33:6823/1946811008","heartbeat_front_addr":"172.21.15.33:6821/1946811008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-10T18:50:38.334630+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-10T18:50:38.120123+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-10T18:50:38.338202+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-10T18:50:49.573 INFO:tasks.ceph.ceph_manager.cluster1:all up! 2023-12-10T18:50:49.574 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2023-12-10T18:50:49.858 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-10T18:50:49.858 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":13,"fsid":"d3c37dcf-9605-4f22-911b-14b903774172","created":"2023-12-10T18:50:31.918913+0000","modified":"2023-12-10T18:50:48.869027+0000","last_up_change":"2023-12-10T18:50:42.021651+0000","last_in_change":"2023-12-10T18:50:32.830957+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-10T18:50:42.925167+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-10T18:50:47.511893+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cade80c6-664a-4a0a-8298-0c400418aad4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":223693307},{"type":"v1","addr":"172.21.15.33:6809","nonce":223693307}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":223693307},{"type":"v1","addr":"172.21.15.33:6811","nonce":223693307}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":223693307},{"type":"v1","addr":"172.21.15.33:6815","nonce":223693307}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":223693307},{"type":"v1","addr":"172.21.15.33:6813","nonce":223693307}]},"public_addr":"172.21.15.33:6809/223693307","cluster_addr":"172.21.15.33:6811/223693307","heartbeat_back_addr":"172.21.15.33:6815/223693307","heartbeat_front_addr":"172.21.15.33:6813/223693307","state":["exists","up"]},{"osd":1,"uuid":"90ff3f91-f341-41a0-82fb-67b8ffe22dc5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":3180911142},{"type":"v1","addr":"172.21.15.33:6801","nonce":3180911142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":3180911142},{"type":"v1","addr":"172.21.15.33:6803","nonce":3180911142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":3180911142},{"type":"v1","addr":"172.21.15.33:6807","nonce":3180911142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":3180911142},{"type":"v1","addr":"172.21.15.33:6805","nonce":3180911142}]},"public_addr":"172.21.15.33:6801/3180911142","cluster_addr":"172.21.15.33:6803/3180911142","heartbeat_back_addr":"172.21.15.33:6807/3180911142","heartbeat_front_addr":"172.21.15.33:6805/3180911142","state":["exists","up"]},{"osd":2,"uuid":"50c019c1-d5ef-4bd8-9acf-f6ae6b3f538a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":1946811008},{"type":"v1","addr":"172.21.15.33:6817","nonce":1946811008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":1946811008},{"type":"v1","addr":"172.21.15.33:6819","nonce":1946811008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":1946811008},{"type":"v1","addr":"172.21.15.33:6823","nonce":1946811008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":1946811008},{"type":"v1","addr":"172.21.15.33:6821","nonce":1946811008}]},"public_addr":"172.21.15.33:6817/1946811008","cluster_addr":"172.21.15.33:6819/1946811008","heartbeat_back_addr":"172.21.15.33:6823/1946811008","heartbeat_front_addr":"172.21.15.33:6821/1946811008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-10T18:50:38.334630+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-10T18:50:38.120123+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-10T18:50:38.338202+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-10T18:50:49.877 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 flush_pg_stats 2023-12-10T18:50:49.877 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 flush_pg_stats 2023-12-10T18:50:49.877 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 flush_pg_stats 2023-12-10T18:50:50.033 INFO:teuthology.orchestra.run.smithi033.stdout:34359738372 2023-12-10T18:50:50.034 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2023-12-10T18:50:50.040 INFO:teuthology.orchestra.run.smithi033.stdout:34359738374 2023-12-10T18:50:50.040 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2023-12-10T18:50:50.043 INFO:teuthology.orchestra.run.smithi033.stdout:34359738374 2023-12-10T18:50:50.043 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2023-12-10T18:50:50.336 INFO:teuthology.orchestra.run.smithi033.stdout:34359738371 2023-12-10T18:50:50.353 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738372 got 34359738371 for osd.1 2023-12-10T18:50:50.373 INFO:teuthology.orchestra.run.smithi033.stdout:34359738373 2023-12-10T18:50:50.390 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738374 got 34359738373 for osd.0 2023-12-10T18:50:50.397 INFO:teuthology.orchestra.run.smithi033.stdout:34359738373 2023-12-10T18:50:50.414 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738374 got 34359738373 for osd.2 2023-12-10T18:50:51.355 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2023-12-10T18:50:51.391 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2023-12-10T18:50:51.415 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2023-12-10T18:50:51.662 INFO:teuthology.orchestra.run.smithi033.stdout:34359738372 2023-12-10T18:50:51.680 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738372 got 34359738372 for osd.1 2023-12-10T18:50:51.680 DEBUG:teuthology.parallel:result is None 2023-12-10T18:50:51.693 INFO:teuthology.orchestra.run.smithi033.stdout:34359738374 2023-12-10T18:50:51.710 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738374 got 34359738374 for osd.0 2023-12-10T18:50:51.711 DEBUG:teuthology.parallel:result is None 2023-12-10T18:50:51.734 INFO:teuthology.orchestra.run.smithi033.stdout:34359738374 2023-12-10T18:50:51.750 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738374 got 34359738374 for osd.2 2023-12-10T18:50:51.751 DEBUG:teuthology.parallel:result is None 2023-12-10T18:50:51.751 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2023-12-10T18:50:51.751 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2023-12-10T18:50:52.076 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-10T18:50:52.076 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2023-12-10T18:50:52.092 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2023-12-10T18:50:50.852565+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82552,"kb_used_data":1492,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032968,"statfs":{"total":289910292480,"available":289825759232,"internally_reserved":0,"allocated":1527808,"data_stored":1273960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000582"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-10T18:50:48.898686+0000","last_change":"2023-12-10T18:50:48.898686+0000","last_active":"2023-12-10T18:50:48.898686+0000","last_peered":"2023-12-10T18:50:48.898686+0000","last_clean":"2023-12-10T18:50:48.898686+0000","last_became_active":"2023-12-10T18:50:48.898256+0000","last_became_peered":"2023-12-10T18:50:48.898256+0000","last_unstale":"2023-12-10T18:50:48.898686+0000","last_undegraded":"2023-12-10T18:50:48.898686+0000","last_fullsized":"2023-12-10T18:50:48.898686+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-10T18:50:47.857924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-10T18:50:47.857924+0000","last_clean_scrub_stamp":"2023-12-10T18:50:47.857924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-11T19:05:38.409743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-10T18:50:48.896207+0000","last_change":"2023-12-10T18:50:48.896207+0000","last_active":"2023-12-10T18:50:48.896207+0000","last_peered":"2023-12-10T18:50:48.896207+0000","last_clean":"2023-12-10T18:50:48.896207+0000","last_became_active":"2023-12-10T18:50:48.895866+0000","last_became_peered":"2023-12-10T18:50:48.895866+0000","last_unstale":"2023-12-10T18:50:48.896207+0000","last_undegraded":"2023-12-10T18:50:48.896207+0000","last_fullsized":"2023-12-10T18:50:48.896207+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-10T18:50:47.857924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-10T18:50:47.857924+0000","last_clean_scrub_stamp":"2023-12-10T18:50:47.857924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-11T20:27:45.665578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-10T18:50:48.899098+0000","last_change":"2023-12-10T18:50:48.899098+0000","last_active":"2023-12-10T18:50:48.899098+0000","last_peered":"2023-12-10T18:50:48.899098+0000","last_clean":"2023-12-10T18:50:48.899098+0000","last_became_active":"2023-12-10T18:50:48.898666+0000","last_became_peered":"2023-12-10T18:50:48.898666+0000","last_unstale":"2023-12-10T18:50:48.899098+0000","last_undegraded":"2023-12-10T18:50:48.899098+0000","last_fullsized":"2023-12-10T18:50:48.899098+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-10T18:50:47.857924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-10T18:50:47.857924+0000","last_clean_scrub_stamp":"2023-12-10T18:50:47.857924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-11T21:22:41.809198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-10T18:50:48.898893+0000","last_change":"2023-12-10T18:50:48.898893+0000","last_active":"2023-12-10T18:50:48.898893+0000","last_peered":"2023-12-10T18:50:48.898893+0000","last_clean":"2023-12-10T18:50:48.898893+0000","last_became_active":"2023-12-10T18:50:48.898461+0000","last_became_peered":"2023-12-10T18:50:48.898461+0000","last_unstale":"2023-12-10T18:50:48.898893+0000","last_undegraded":"2023-12-10T18:50:48.898893+0000","last_fullsized":"2023-12-10T18:50:48.898893+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-10T18:50:47.857924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-10T18:50:47.857924+0000","last_clean_scrub_stamp":"2023-12-10T18:50:47.857924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T06:24:10.531464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-10T18:50:48.897314+0000","last_change":"2023-12-10T18:50:48.897314+0000","last_active":"2023-12-10T18:50:48.897314+0000","last_peered":"2023-12-10T18:50:48.897314+0000","last_clean":"2023-12-10T18:50:48.897314+0000","last_became_active":"2023-12-10T18:50:48.896523+0000","last_became_peered":"2023-12-10T18:50:48.896523+0000","last_unstale":"2023-12-10T18:50:48.897314+0000","last_undegraded":"2023-12-10T18:50:48.897314+0000","last_fullsized":"2023-12-10T18:50:48.897314+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-10T18:50:47.857924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-10T18:50:47.857924+0000","last_clean_scrub_stamp":"2023-12-10T18:50:47.857924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T06:00:31.566236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-10T18:50:49.890577+0000","last_change":"2023-12-10T18:50:48.897483+0000","last_active":"2023-12-10T18:50:49.890577+0000","last_peered":"2023-12-10T18:50:49.890577+0000","last_clean":"2023-12-10T18:50:49.890577+0000","last_became_active":"2023-12-10T18:50:48.897076+0000","last_became_peered":"2023-12-10T18:50:48.897076+0000","last_unstale":"2023-12-10T18:50:49.890577+0000","last_undegraded":"2023-12-10T18:50:49.890577+0000","last_fullsized":"2023-12-10T18:50:49.890577+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-10T18:50:47.857924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-10T18:50:47.857924+0000","last_clean_scrub_stamp":"2023-12-10T18:50:47.857924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-11T23:55:38.397931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-10T18:50:49.890597+0000","last_change":"2023-12-10T18:50:48.900764+0000","last_active":"2023-12-10T18:50:49.890597+0000","last_peered":"2023-12-10T18:50:49.890597+0000","last_clean":"2023-12-10T18:50:49.890597+0000","last_became_active":"2023-12-10T18:50:48.900445+0000","last_became_peered":"2023-12-10T18:50:48.900445+0000","last_unstale":"2023-12-10T18:50:49.890597+0000","last_undegraded":"2023-12-10T18:50:49.890597+0000","last_fullsized":"2023-12-10T18:50:49.890597+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-10T18:50:47.857924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-10T18:50:47.857924+0000","last_clean_scrub_stamp":"2023-12-10T18:50:47.857924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-11T23:35:52.781690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-10T18:50:48.896301+0000","last_change":"2023-12-10T18:50:48.896301+0000","last_active":"2023-12-10T18:50:48.896301+0000","last_peered":"2023-12-10T18:50:48.896301+0000","last_clean":"2023-12-10T18:50:48.896301+0000","last_became_active":"2023-12-10T18:50:48.895871+0000","last_became_peered":"2023-12-10T18:50:48.895871+0000","last_unstale":"2023-12-10T18:50:48.896301+0000","last_undegraded":"2023-12-10T18:50:48.896301+0000","last_fullsized":"2023-12-10T18:50:48.896301+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-10T18:50:47.857924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-10T18:50:47.857924+0000","last_clean_scrub_stamp":"2023-12-10T18:50:47.857924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T04:47:17.967877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":140,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-10T18:50:47.920790+0000","last_change":"2023-12-10T18:50:44.250843+0000","last_active":"2023-12-10T18:50:47.920790+0000","last_peered":"2023-12-10T18:50:47.920790+0000","last_clean":"2023-12-10T18:50:47.920790+0000","last_became_active":"2023-12-10T18:50:44.248665+0000","last_became_peered":"2023-12-10T18:50:44.248665+0000","last_unstale":"2023-12-10T18:50:47.920790+0000","last_undegraded":"2023-12-10T18:50:47.920790+0000","last_fullsized":"2023-12-10T18:50:47.920790+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-10T18:50:43.023682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-10T18:50:43.023682+0000","last_clean_scrub_stamp":"2023-12-10T18:50:43.023682+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T04:25:51.035619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27132,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344708,"statfs":{"total":96636764160,"available":96608980992,"internally_reserved":0,"allocated":114688,"data_stored":31331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54900000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66800000000000004}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":708608,"data_stored":621699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30299999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27708,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344132,"statfs":{"total":96636764160,"available":96608391168,"internally_reserved":0,"allocated":704512,"data_stored":620930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-10T18:50:52.093 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2023-12-10T18:50:52.374 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-10T18:50:52.374 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2023-12-10T18:50:52.391 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2023-12-10T18:50:50.852565+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82552,"kb_used_data":1492,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032968,"statfs":{"total":289910292480,"available":289825759232,"internally_reserved":0,"allocated":1527808,"data_stored":1273960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000582"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-10T18:50:48.898686+0000","last_change":"2023-12-10T18:50:48.898686+0000","last_active":"2023-12-10T18:50:48.898686+0000","last_peered":"2023-12-10T18:50:48.898686+0000","last_clean":"2023-12-10T18:50:48.898686+0000","last_became_active":"2023-12-10T18:50:48.898256+0000","last_became_peered":"2023-12-10T18:50:48.898256+0000","last_unstale":"2023-12-10T18:50:48.898686+0000","last_undegraded":"2023-12-10T18:50:48.898686+0000","last_fullsized":"2023-12-10T18:50:48.898686+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-10T18:50:47.857924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-10T18:50:47.857924+0000","last_clean_scrub_stamp":"2023-12-10T18:50:47.857924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-11T19:05:38.409743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-10T18:50:48.896207+0000","last_change":"2023-12-10T18:50:48.896207+0000","last_active":"2023-12-10T18:50:48.896207+0000","last_peered":"2023-12-10T18:50:48.896207+0000","last_clean":"2023-12-10T18:50:48.896207+0000","last_became_active":"2023-12-10T18:50:48.895866+0000","last_became_peered":"2023-12-10T18:50:48.895866+0000","last_unstale":"2023-12-10T18:50:48.896207+0000","last_undegraded":"2023-12-10T18:50:48.896207+0000","last_fullsized":"2023-12-10T18:50:48.896207+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-10T18:50:47.857924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-10T18:50:47.857924+0000","last_clean_scrub_stamp":"2023-12-10T18:50:47.857924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-11T20:27:45.665578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-10T18:50:48.899098+0000","last_change":"2023-12-10T18:50:48.899098+0000","last_active":"2023-12-10T18:50:48.899098+0000","last_peered":"2023-12-10T18:50:48.899098+0000","last_clean":"2023-12-10T18:50:48.899098+0000","last_became_active":"2023-12-10T18:50:48.898666+0000","last_became_peered":"2023-12-10T18:50:48.898666+0000","last_unstale":"2023-12-10T18:50:48.899098+0000","last_undegraded":"2023-12-10T18:50:48.899098+0000","last_fullsized":"2023-12-10T18:50:48.899098+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-10T18:50:47.857924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-10T18:50:47.857924+0000","last_clean_scrub_stamp":"2023-12-10T18:50:47.857924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-11T21:22:41.809198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-10T18:50:48.898893+0000","last_change":"2023-12-10T18:50:48.898893+0000","last_active":"2023-12-10T18:50:48.898893+0000","last_peered":"2023-12-10T18:50:48.898893+0000","last_clean":"2023-12-10T18:50:48.898893+0000","last_became_active":"2023-12-10T18:50:48.898461+0000","last_became_peered":"2023-12-10T18:50:48.898461+0000","last_unstale":"2023-12-10T18:50:48.898893+0000","last_undegraded":"2023-12-10T18:50:48.898893+0000","last_fullsized":"2023-12-10T18:50:48.898893+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-10T18:50:47.857924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-10T18:50:47.857924+0000","last_clean_scrub_stamp":"2023-12-10T18:50:47.857924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T06:24:10.531464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-10T18:50:48.897314+0000","last_change":"2023-12-10T18:50:48.897314+0000","last_active":"2023-12-10T18:50:48.897314+0000","last_peered":"2023-12-10T18:50:48.897314+0000","last_clean":"2023-12-10T18:50:48.897314+0000","last_became_active":"2023-12-10T18:50:48.896523+0000","last_became_peered":"2023-12-10T18:50:48.896523+0000","last_unstale":"2023-12-10T18:50:48.897314+0000","last_undegraded":"2023-12-10T18:50:48.897314+0000","last_fullsized":"2023-12-10T18:50:48.897314+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-10T18:50:47.857924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-10T18:50:47.857924+0000","last_clean_scrub_stamp":"2023-12-10T18:50:47.857924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T06:00:31.566236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-10T18:50:49.890577+0000","last_change":"2023-12-10T18:50:48.897483+0000","last_active":"2023-12-10T18:50:49.890577+0000","last_peered":"2023-12-10T18:50:49.890577+0000","last_clean":"2023-12-10T18:50:49.890577+0000","last_became_active":"2023-12-10T18:50:48.897076+0000","last_became_peered":"2023-12-10T18:50:48.897076+0000","last_unstale":"2023-12-10T18:50:49.890577+0000","last_undegraded":"2023-12-10T18:50:49.890577+0000","last_fullsized":"2023-12-10T18:50:49.890577+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-10T18:50:47.857924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-10T18:50:47.857924+0000","last_clean_scrub_stamp":"2023-12-10T18:50:47.857924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-11T23:55:38.397931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-10T18:50:49.890597+0000","last_change":"2023-12-10T18:50:48.900764+0000","last_active":"2023-12-10T18:50:49.890597+0000","last_peered":"2023-12-10T18:50:49.890597+0000","last_clean":"2023-12-10T18:50:49.890597+0000","last_became_active":"2023-12-10T18:50:48.900445+0000","last_became_peered":"2023-12-10T18:50:48.900445+0000","last_unstale":"2023-12-10T18:50:49.890597+0000","last_undegraded":"2023-12-10T18:50:49.890597+0000","last_fullsized":"2023-12-10T18:50:49.890597+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-10T18:50:47.857924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-10T18:50:47.857924+0000","last_clean_scrub_stamp":"2023-12-10T18:50:47.857924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-11T23:35:52.781690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-10T18:50:48.896301+0000","last_change":"2023-12-10T18:50:48.896301+0000","last_active":"2023-12-10T18:50:48.896301+0000","last_peered":"2023-12-10T18:50:48.896301+0000","last_clean":"2023-12-10T18:50:48.896301+0000","last_became_active":"2023-12-10T18:50:48.895871+0000","last_became_peered":"2023-12-10T18:50:48.895871+0000","last_unstale":"2023-12-10T18:50:48.896301+0000","last_undegraded":"2023-12-10T18:50:48.896301+0000","last_fullsized":"2023-12-10T18:50:48.896301+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-10T18:50:47.857924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-10T18:50:47.857924+0000","last_clean_scrub_stamp":"2023-12-10T18:50:47.857924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T04:47:17.967877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":140,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-10T18:50:47.920790+0000","last_change":"2023-12-10T18:50:44.250843+0000","last_active":"2023-12-10T18:50:47.920790+0000","last_peered":"2023-12-10T18:50:47.920790+0000","last_clean":"2023-12-10T18:50:47.920790+0000","last_became_active":"2023-12-10T18:50:44.248665+0000","last_became_peered":"2023-12-10T18:50:44.248665+0000","last_unstale":"2023-12-10T18:50:47.920790+0000","last_undegraded":"2023-12-10T18:50:47.920790+0000","last_fullsized":"2023-12-10T18:50:47.920790+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-10T18:50:43.023682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-10T18:50:43.023682+0000","last_clean_scrub_stamp":"2023-12-10T18:50:43.023682+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T04:25:51.035619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27132,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344708,"statfs":{"total":96636764160,"available":96608980992,"internally_reserved":0,"allocated":114688,"data_stored":31331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54900000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66800000000000004}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":708608,"data_stored":621699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30299999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27708,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344132,"statfs":{"total":96636764160,"available":96608391168,"internally_reserved":0,"allocated":704512,"data_stored":620930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-10T18:50:52.391 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2023-12-10T18:50:52.392 INFO:tasks.ceph:Waiting until ceph cluster cluster1 is healthy... 2023-12-10T18:50:52.392 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy 2023-12-10T18:50:52.392 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 health --format=json 2023-12-10T18:50:52.712 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-10T18:50:52.713 INFO:teuthology.orchestra.run.smithi033.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-12-10T18:50:52.731 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy done 2023-12-10T18:50:52.732 INFO:teuthology.run_tasks:Running task ceph... 2023-12-10T18:50:52.743 INFO:tasks.ceph:Creating ceph cluster cluster2... 2023-12-10T18:50:52.743 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-12-10T18:50:52.743 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/dis-2023-12-10_16:10:42-rbd-wip-dis-testing-1-distro-default-smithi/7486362', 'branch': 'wip-dis-testing-1', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-stupid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-stress-workunit-minimum}', 'email': 'idryomov@gmail.com', 'first_in_suite': False, 'job_id': '7486362', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'dis-2023-12-10_16:10:42-rbd-wip-dis-testing-1-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-dis-testing-1'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '3d2db44fa991bc355074aa9b7647f63b77201fef'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '3d2db44fa991bc355074aa9b7647f63b77201fef'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-dis-testing-1', 'sha1': '3d2db44fa991bc355074aa9b7647f63b77201fef'}}, 'owner': 'scheduled_dis@teuthology', 'priority': 74, 'repo': 'https://github.com/ceph/ceph-ci', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 7033, 'sha1': '3d2db44fa991bc355074aa9b7647f63b77201fef', 'sleep_before_teardown': 0, 'subset': '91/99', 'suite': 'rbd', 'suite_branch': 'wip-dis-testing-1', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci', 'suite_sha1': '3d2db44fa991bc355074aa9b7647f63b77201fef', 'targets': {'smithi033.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCg4fpIf9BgAE0K5vY3eaCYNnAdpFkPYg83UNzpXJx+VdnMglbkcDNn9oNHV0h8GlQ68zOzCcQCAjDqUW/H3wNI=', 'smithi076.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDz6mTNSmU7E4IwEUFsmyAfIf0TTvzaqidmq2OKROs5pSoYL6ng2SXq1yuh1NB7AMqvNCBdduJ5kk6F5IVYNexs='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': '3d2db44fa991bc355074aa9b7647f63b77201fef'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '3d2db44fa991bc355074aa9b7647f63b77201fef'}}, {'ceph': {'cluster': 'cluster2', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '3d2db44fa991bc355074aa9b7647f63b77201fef'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_MODE': 'snapshot', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'f2de108beb0a6dde7f3e6c96039071828a1f6269', 'timestamp': '2023-12-10_16:10:42', 'tube': 'smithi', 'user': 'dis', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.232157'} 2023-12-10T18:50:52.744 DEBUG:teuthology.orchestra.run.smithi033:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2023-12-10T18:50:52.746 DEBUG:teuthology.orchestra.run.smithi076:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2023-12-10T18:50:52.754 DEBUG:teuthology.orchestra.run.smithi033:> sudo install -d -m0777 -- /var/run/ceph 2023-12-10T18:50:52.796 DEBUG:teuthology.orchestra.run.smithi076:> sudo install -d -m0777 -- /var/run/ceph 2023-12-10T18:50:52.811 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2023-12-10T18:50:52.811 DEBUG:teuthology.orchestra.run.smithi076:> dd if=/scratch_devs of=/dev/stdout 2023-12-10T18:50:52.858 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-10T18:50:52.858 DEBUG:teuthology.orchestra.run.smithi076:> stat /dev/vg_nvme/lv_1 2023-12-10T18:50:52.906 INFO:teuthology.orchestra.run.smithi076.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-10T18:50:52.907 INFO:teuthology.orchestra.run.smithi076.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-10T18:50:52.907 INFO:teuthology.orchestra.run.smithi076.stdout:Device: 5h/5d Inode: 890 Links: 1 2023-12-10T18:50:52.907 INFO:teuthology.orchestra.run.smithi076.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-10T18:50:52.907 INFO:teuthology.orchestra.run.smithi076.stdout:Access: 2023-12-10 18:44:55.072627660 +0000 2023-12-10T18:50:52.907 INFO:teuthology.orchestra.run.smithi076.stdout:Modify: 2023-12-10 18:44:54.720630803 +0000 2023-12-10T18:50:52.907 INFO:teuthology.orchestra.run.smithi076.stdout:Change: 2023-12-10 18:44:54.720630803 +0000 2023-12-10T18:50:52.907 INFO:teuthology.orchestra.run.smithi076.stdout: Birth: - 2023-12-10T18:50:52.908 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-10T18:50:52.960 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records in 2023-12-10T18:50:52.961 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records out 2023-12-10T18:50:52.961 INFO:teuthology.orchestra.run.smithi076.stderr:512 bytes copied, 0.000401816 s, 1.3 MB/s 2023-12-10T18:50:52.962 DEBUG:teuthology.orchestra.run.smithi076:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-10T18:50:53.011 DEBUG:teuthology.orchestra.run.smithi076:> stat /dev/vg_nvme/lv_2 2023-12-10T18:50:53.058 INFO:teuthology.orchestra.run.smithi076.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-10T18:50:53.059 INFO:teuthology.orchestra.run.smithi076.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-10T18:50:53.059 INFO:teuthology.orchestra.run.smithi076.stdout:Device: 5h/5d Inode: 907 Links: 1 2023-12-10T18:50:53.059 INFO:teuthology.orchestra.run.smithi076.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-10T18:50:53.059 INFO:teuthology.orchestra.run.smithi076.stdout:Access: 2023-12-10 18:44:55.648622515 +0000 2023-12-10T18:50:53.059 INFO:teuthology.orchestra.run.smithi076.stdout:Modify: 2023-12-10 18:44:55.296625658 +0000 2023-12-10T18:50:53.059 INFO:teuthology.orchestra.run.smithi076.stdout:Change: 2023-12-10 18:44:55.296625658 +0000 2023-12-10T18:50:53.060 INFO:teuthology.orchestra.run.smithi076.stdout: Birth: - 2023-12-10T18:50:53.060 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-10T18:50:53.112 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records in 2023-12-10T18:50:53.112 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records out 2023-12-10T18:50:53.112 INFO:teuthology.orchestra.run.smithi076.stderr:512 bytes copied, 0.000392261 s, 1.3 MB/s 2023-12-10T18:50:53.114 DEBUG:teuthology.orchestra.run.smithi076:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-10T18:50:53.163 DEBUG:teuthology.orchestra.run.smithi076:> stat /dev/vg_nvme/lv_3 2023-12-10T18:50:53.210 INFO:teuthology.orchestra.run.smithi076.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-10T18:50:53.210 INFO:teuthology.orchestra.run.smithi076.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-10T18:50:53.210 INFO:teuthology.orchestra.run.smithi076.stdout:Device: 5h/5d Inode: 925 Links: 1 2023-12-10T18:50:53.210 INFO:teuthology.orchestra.run.smithi076.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-10T18:50:53.211 INFO:teuthology.orchestra.run.smithi076.stdout:Access: 2023-12-10 18:44:56.228617334 +0000 2023-12-10T18:50:53.211 INFO:teuthology.orchestra.run.smithi076.stdout:Modify: 2023-12-10 18:44:55.880620443 +0000 2023-12-10T18:50:53.211 INFO:teuthology.orchestra.run.smithi076.stdout:Change: 2023-12-10 18:44:55.880620443 +0000 2023-12-10T18:50:53.211 INFO:teuthology.orchestra.run.smithi076.stdout: Birth: - 2023-12-10T18:50:53.211 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-10T18:50:53.263 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records in 2023-12-10T18:50:53.263 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records out 2023-12-10T18:50:53.264 INFO:teuthology.orchestra.run.smithi076.stderr:512 bytes copied, 0.000310097 s, 1.7 MB/s 2023-12-10T18:50:53.265 DEBUG:teuthology.orchestra.run.smithi076:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-10T18:50:53.316 DEBUG:teuthology.orchestra.run.smithi076:> stat /dev/vg_nvme/lv_4 2023-12-10T18:50:53.363 INFO:teuthology.orchestra.run.smithi076.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-10T18:50:53.363 INFO:teuthology.orchestra.run.smithi076.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-10T18:50:53.363 INFO:teuthology.orchestra.run.smithi076.stdout:Device: 5h/5d Inode: 943 Links: 1 2023-12-10T18:50:53.363 INFO:teuthology.orchestra.run.smithi076.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-10T18:50:53.363 INFO:teuthology.orchestra.run.smithi076.stdout:Access: 2023-12-10 18:44:56.820612046 +0000 2023-12-10T18:50:53.364 INFO:teuthology.orchestra.run.smithi076.stdout:Modify: 2023-12-10 18:44:56.472615155 +0000 2023-12-10T18:50:53.364 INFO:teuthology.orchestra.run.smithi076.stdout:Change: 2023-12-10 18:44:56.472615155 +0000 2023-12-10T18:50:53.364 INFO:teuthology.orchestra.run.smithi076.stdout: Birth: - 2023-12-10T18:50:53.364 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-10T18:50:53.417 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records in 2023-12-10T18:50:53.417 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records out 2023-12-10T18:50:53.417 INFO:teuthology.orchestra.run.smithi076.stderr:512 bytes copied, 0.000387954 s, 1.3 MB/s 2023-12-10T18:50:53.419 DEBUG:teuthology.orchestra.run.smithi076:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-10T18:50:53.468 INFO:tasks.ceph:osd dev map: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-10T18:50:53.468 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi076.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2023-12-10T18:50:53.468 INFO:tasks.ceph:Generating config... 2023-12-10T18:50:53.469 INFO:tasks.ceph:[client] debug rbd = 20 2023-12-10T18:50:53.470 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2023-12-10T18:50:53.470 INFO:tasks.ceph:[client] log to stderr = False 2023-12-10T18:50:53.470 INFO:tasks.ceph:[client] rbd default features = 125 2023-12-10T18:50:53.470 INFO:tasks.ceph:[client] rbd mirror image policy type = simple 2023-12-10T18:50:53.470 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-10T18:50:53.470 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-10T18:50:53.470 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-10T18:50:53.470 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-10T18:50:53.470 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-10T18:50:53.470 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-10T18:50:53.470 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-10T18:50:53.470 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-10T18:50:53.470 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-12-10T18:50:53.471 INFO:tasks.ceph:[global] mon warn on pool no app = False 2023-12-10T18:50:53.471 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-12-10T18:50:53.471 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-12-10T18:50:53.471 INFO:tasks.ceph:[mgr] debug ms = 1 2023-12-10T18:50:53.471 INFO:tasks.ceph:[mon] debug mon = 20 2023-12-10T18:50:53.471 INFO:tasks.ceph:[mon] debug ms = 1 2023-12-10T18:50:53.471 INFO:tasks.ceph:[mon] debug paxos = 20 2023-12-10T18:50:53.471 INFO:tasks.ceph:[osd] bdev async discard = True 2023-12-10T18:50:53.471 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-12-10T18:50:53.471 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2023-12-10T18:50:53.472 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-12-10T18:50:53.472 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-12-10T18:50:53.472 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-12-10T18:50:53.472 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-12-10T18:50:53.472 INFO:tasks.ceph:[osd] debug ms = 1 2023-12-10T18:50:53.472 INFO:tasks.ceph:[osd] debug osd = 20 2023-12-10T18:50:53.472 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-12-10T18:50:53.473 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-12-10T18:50:53.473 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-12-10T18:50:53.473 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-12-10T18:50:53.473 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-12-10T18:50:53.473 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-12-10T18:50:53.473 INFO:tasks.ceph:Setting up cluster2.mon.a... 2023-12-10T18:50:53.474 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster2.keyring 2023-12-10T18:50:53.540 INFO:teuthology.orchestra.run.smithi076.stdout:creating /etc/ceph/cluster2.keyring 2023-12-10T18:50:53.543 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster2.keyring 2023-12-10T18:50:53.618 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2023-12-10T18:50:53.673 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster2.mon.a', '172.21.15.76')] 2023-12-10T18:50:53.674 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.76', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2023-12-10T18:50:53.675 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2023-12-10T18:50:53.675 DEBUG:teuthology.orchestra.run.smithi076:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-12-10T18:50:53.722 DEBUG:teuthology.orchestra.run.smithi076:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.76 --print /home/ubuntu/cephtest/cluster2.monmap 2023-12-10T18:50:53.801 INFO:teuthology.orchestra.run.smithi076.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster2.monmap 2023-12-10T18:50:53.802 INFO:teuthology.orchestra.run.smithi076.stdout:monmaptool: generated fsid af537260-6e4d-44cb-9e7e-e1ba6e0ece8e 2023-12-10T18:50:53.802 INFO:teuthology.orchestra.run.smithi076.stdout:setting min_mon_release = pacific 2023-12-10T18:50:53.802 INFO:teuthology.orchestra.run.smithi076.stdout:epoch 0 2023-12-10T18:50:53.802 INFO:teuthology.orchestra.run.smithi076.stdout:fsid af537260-6e4d-44cb-9e7e-e1ba6e0ece8e 2023-12-10T18:50:53.802 INFO:teuthology.orchestra.run.smithi076.stdout:last_changed 2023-12-10T18:50:53.799551+0000 2023-12-10T18:50:53.802 INFO:teuthology.orchestra.run.smithi076.stdout:created 2023-12-10T18:50:53.799551+0000 2023-12-10T18:50:53.803 INFO:teuthology.orchestra.run.smithi076.stdout:min_mon_release 16 (pacific) 2023-12-10T18:50:53.803 INFO:teuthology.orchestra.run.smithi076.stdout:election_strategy: 1 2023-12-10T18:50:53.803 INFO:teuthology.orchestra.run.smithi076.stdout:0: [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] mon.a 2023-12-10T18:50:53.803 INFO:teuthology.orchestra.run.smithi076.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster2.monmap (1 monitors) 2023-12-10T18:50:53.805 DEBUG:teuthology.orchestra.run.smithi076:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-12-10T18:50:53.850 INFO:tasks.ceph:Writing /etc/ceph/cluster2.conf for FSID af537260-6e4d-44cb-9e7e-e1ba6e0ece8e... 2023-12-10T18:50:53.851 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2023-12-10T18:50:53.854 DEBUG:teuthology.orchestra.run.smithi076:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2023-12-10T18:50:53.896 INFO:teuthology.orchestra.run.smithi033.stdout:[global] 2023-12-10T18:50:53.896 INFO:teuthology.orchestra.run.smithi033.stdout: chdir = "" 2023-12-10T18:50:53.896 INFO:teuthology.orchestra.run.smithi033.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-10T18:50:53.896 INFO:teuthology.orchestra.run.smithi033.stdout: auth supported = cephx 2023-12-10T18:50:53.896 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-10T18:50:53.896 INFO:teuthology.orchestra.run.smithi033.stdout: filestore xattr use omap = true 2023-12-10T18:50:53.896 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-10T18:50:53.897 INFO:teuthology.orchestra.run.smithi033.stdout: mon clock drift allowed = 1.000 2023-12-10T18:50:53.897 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-10T18:50:53.897 INFO:teuthology.orchestra.run.smithi033.stdout: osd crush chooseleaf type = 0 2023-12-10T18:50:53.897 INFO:teuthology.orchestra.run.smithi033.stdout: auth debug = true 2023-12-10T18:50:53.897 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-10T18:50:53.897 INFO:teuthology.orchestra.run.smithi033.stdout: ms die on old message = true 2023-12-10T18:50:53.897 INFO:teuthology.orchestra.run.smithi033.stdout: ms die on bug = true 2023-12-10T18:50:53.897 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-10T18:50:53.897 INFO:teuthology.orchestra.run.smithi033.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-10T18:50:53.897 INFO:teuthology.orchestra.run.smithi033.stdout: mon pg warn max object skew = 0 2023-12-10T18:50:53.897 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-10T18:50:53.897 INFO:teuthology.orchestra.run.smithi033.stdout: # disable pg_autoscaler by default for new pools 2023-12-10T18:50:53.897 INFO:teuthology.orchestra.run.smithi033.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-10T18:50:53.897 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-10T18:50:53.898 INFO:teuthology.orchestra.run.smithi033.stdout: osd pool default size = 2 2023-12-10T18:50:53.898 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-10T18:50:53.898 INFO:teuthology.orchestra.run.smithi033.stdout: mon osd allow primary affinity = true 2023-12-10T18:50:53.898 INFO:teuthology.orchestra.run.smithi033.stdout: mon osd allow pg remap = true 2023-12-10T18:50:53.898 INFO:teuthology.orchestra.run.smithi033.stdout: mon warn on legacy crush tunables = false 2023-12-10T18:50:53.899 INFO:teuthology.orchestra.run.smithi033.stdout: mon warn on crush straw calc version zero = false 2023-12-10T18:50:53.899 INFO:teuthology.orchestra.run.smithi033.stdout: mon warn on no sortbitwise = false 2023-12-10T18:50:53.899 INFO:teuthology.orchestra.run.smithi033.stdout: mon warn on osd down out interval zero = false 2023-12-10T18:50:53.899 INFO:teuthology.orchestra.run.smithi033.stdout: mon warn on too few osds = false 2023-12-10T18:50:53.899 INFO:teuthology.orchestra.run.smithi033.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-10T18:50:53.899 INFO:teuthology.orchestra.run.smithi033.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-10T18:50:53.899 INFO:teuthology.orchestra.run.smithi033.stdout: mon_allow_pool_size_one = true 2023-12-10T18:50:53.900 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-10T18:50:53.900 INFO:teuthology.orchestra.run.smithi033.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-10T18:50:53.900 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-10T18:50:53.900 INFO:teuthology.orchestra.run.smithi033.stdout: osd default data pool replay window = 5 2023-12-10T18:50:53.900 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-10T18:50:53.900 INFO:teuthology.orchestra.run.smithi033.stdout: mon allow pool delete = true 2023-12-10T18:50:53.900 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-10T18:50:53.900 INFO:teuthology.orchestra.run.smithi033.stdout: mon cluster log file level = debug 2023-12-10T18:50:53.900 INFO:teuthology.orchestra.run.smithi033.stdout: debug asserts on shutdown = true 2023-12-10T18:50:53.901 INFO:teuthology.orchestra.run.smithi033.stdout: mon health detail to clog = false 2023-12-10T18:50:53.901 INFO:teuthology.orchestra.run.smithi033.stdout: mon host = 172.21.15.76 2023-12-10T18:50:53.901 INFO:teuthology.orchestra.run.smithi033.stdout: mon client directed command retry = 5 2023-12-10T18:50:53.901 INFO:teuthology.orchestra.run.smithi033.stdout: mon warn on pool no app = False 2023-12-10T18:50:53.901 INFO:teuthology.orchestra.run.smithi033.stdout: ms inject socket failures = 5000 2023-12-10T18:50:53.901 INFO:teuthology.orchestra.run.smithi033.stdout: fsid = af537260-6e4d-44cb-9e7e-e1ba6e0ece8e 2023-12-10T18:50:53.901 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-10T18:50:53.901 INFO:teuthology.orchestra.run.smithi033.stdout:[osd] 2023-12-10T18:50:53.901 INFO:teuthology.orchestra.run.smithi033.stdout: osd journal size = 100 2023-12-10T18:50:53.902 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-10T18:50:53.902 INFO:teuthology.orchestra.run.smithi033.stdout: osd scrub load threshold = 5.0 2023-12-10T18:50:53.902 INFO:teuthology.orchestra.run.smithi033.stdout: osd scrub max interval = 600 2023-12-10T18:50:53.902 INFO:teuthology.orchestra.run.smithi033.stdout: osd mclock profile = high_recovery_ops 2023-12-10T18:50:53.902 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-10T18:50:53.903 INFO:teuthology.orchestra.run.smithi033.stdout: osd recover clone overlap = true 2023-12-10T18:50:53.903 INFO:teuthology.orchestra.run.smithi033.stdout: osd recovery max chunk = 1048576 2023-12-10T18:50:53.903 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-10T18:50:53.903 INFO:teuthology.orchestra.run.smithi033.stdout: osd debug shutdown = true 2023-12-10T18:50:53.903 INFO:teuthology.orchestra.run.smithi033.stdout: osd debug op order = true 2023-12-10T18:50:53.904 INFO:teuthology.orchestra.run.smithi033.stdout: osd debug verify stray on activate = true 2023-12-10T18:50:53.904 INFO:teuthology.orchestra.run.smithi033.stdout: osd debug trim objects = true 2023-12-10T18:50:53.904 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-10T18:50:53.904 INFO:teuthology.orchestra.run.smithi033.stdout: osd open classes on start = true 2023-12-10T18:50:53.904 INFO:teuthology.orchestra.run.smithi033.stdout: osd debug pg log writeout = true 2023-12-10T18:50:53.904 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-10T18:50:53.904 INFO:teuthology.orchestra.run.smithi033.stdout: osd deep scrub update digest min age = 30 2023-12-10T18:50:53.904 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-10T18:50:53.905 INFO:teuthology.orchestra.run.smithi033.stdout: osd map max advance = 10 2023-12-10T18:50:53.905 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-10T18:50:53.905 INFO:teuthology.orchestra.run.smithi033.stdout: journal zero on create = true 2023-12-10T18:50:53.905 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-10T18:50:53.905 INFO:teuthology.orchestra.run.smithi033.stdout: filestore ondisk finisher threads = 3 2023-12-10T18:50:53.905 INFO:teuthology.orchestra.run.smithi033.stdout: filestore apply finisher threads = 3 2023-12-10T18:50:53.905 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-10T18:50:53.905 INFO:teuthology.orchestra.run.smithi033.stdout: bdev debug aio = true 2023-12-10T18:50:53.906 INFO:teuthology.orchestra.run.smithi033.stdout: osd debug misdirected ops = true 2023-12-10T18:50:53.906 INFO:teuthology.orchestra.run.smithi033.stdout: bdev async discard = True 2023-12-10T18:50:53.906 INFO:teuthology.orchestra.run.smithi033.stdout: bdev enable discard = True 2023-12-10T18:50:53.906 INFO:teuthology.orchestra.run.smithi033.stdout: bluestore allocator = stupid 2023-12-10T18:50:53.906 INFO:teuthology.orchestra.run.smithi033.stdout: bluestore block size = 96636764160 2023-12-10T18:50:53.906 INFO:teuthology.orchestra.run.smithi033.stdout: bluestore fsck on mount = True 2023-12-10T18:50:53.906 INFO:teuthology.orchestra.run.smithi033.stdout: debug bluefs = 1/20 2023-12-10T18:50:53.906 INFO:teuthology.orchestra.run.smithi033.stdout: debug bluestore = 1/20 2023-12-10T18:50:53.906 INFO:teuthology.orchestra.run.smithi033.stdout: debug ms = 1 2023-12-10T18:50:53.907 INFO:teuthology.orchestra.run.smithi033.stdout: debug osd = 20 2023-12-10T18:50:53.907 INFO:teuthology.orchestra.run.smithi033.stdout: debug rocksdb = 4/10 2023-12-10T18:50:53.907 INFO:teuthology.orchestra.run.smithi033.stdout: mon osd backfillfull_ratio = 0.85 2023-12-10T18:50:53.907 INFO:teuthology.orchestra.run.smithi033.stdout: mon osd full ratio = 0.9 2023-12-10T18:50:53.908 INFO:teuthology.orchestra.run.smithi033.stdout: mon osd nearfull ratio = 0.8 2023-12-10T18:50:53.908 INFO:teuthology.orchestra.run.smithi033.stdout: osd failsafe full ratio = 0.95 2023-12-10T18:50:53.908 INFO:teuthology.orchestra.run.smithi033.stdout: osd objectstore = bluestore 2023-12-10T18:50:53.908 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-10T18:50:53.908 INFO:teuthology.orchestra.run.smithi033.stdout:[mgr] 2023-12-10T18:50:53.908 INFO:teuthology.orchestra.run.smithi033.stdout: debug ms = 1 2023-12-10T18:50:53.908 INFO:teuthology.orchestra.run.smithi033.stdout: debug mgr = 20 2023-12-10T18:50:53.909 INFO:teuthology.orchestra.run.smithi033.stdout: debug mon = 20 2023-12-10T18:50:53.909 INFO:teuthology.orchestra.run.smithi033.stdout: debug auth = 20 2023-12-10T18:50:53.909 INFO:teuthology.orchestra.run.smithi033.stdout: mon reweight min pgs per osd = 4 2023-12-10T18:50:53.909 INFO:teuthology.orchestra.run.smithi033.stdout: mon reweight min bytes per osd = 10 2023-12-10T18:50:53.909 INFO:teuthology.orchestra.run.smithi033.stdout: mgr/telemetry/nag = false 2023-12-10T18:50:53.910 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-10T18:50:53.910 INFO:teuthology.orchestra.run.smithi033.stdout:[mon] 2023-12-10T18:50:53.910 INFO:teuthology.orchestra.run.smithi033.stdout: debug ms = 1 2023-12-10T18:50:53.910 INFO:teuthology.orchestra.run.smithi033.stdout: debug mon = 20 2023-12-10T18:50:53.910 INFO:teuthology.orchestra.run.smithi033.stdout: debug paxos = 20 2023-12-10T18:50:53.910 INFO:teuthology.orchestra.run.smithi033.stdout: debug auth = 20 2023-12-10T18:50:53.910 INFO:teuthology.orchestra.run.smithi033.stdout: mon data avail warn = 5 2023-12-10T18:50:53.910 INFO:teuthology.orchestra.run.smithi033.stdout: mon mgr mkfs grace = 240 2023-12-10T18:50:53.911 INFO:teuthology.orchestra.run.smithi033.stdout: mon reweight min pgs per osd = 4 2023-12-10T18:50:53.911 INFO:teuthology.orchestra.run.smithi033.stdout: mon osd reporter subtree level = osd 2023-12-10T18:50:53.911 INFO:teuthology.orchestra.run.smithi033.stdout: mon osd prime pg temp = true 2023-12-10T18:50:53.911 INFO:teuthology.orchestra.run.smithi033.stdout: mon reweight min bytes per osd = 10 2023-12-10T18:50:53.911 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-10T18:50:53.911 INFO:teuthology.orchestra.run.smithi033.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-10T18:50:53.911 INFO:teuthology.orchestra.run.smithi033.stdout: auth mon ticket ttl = 660 # 11m 2023-12-10T18:50:53.911 INFO:teuthology.orchestra.run.smithi033.stdout: auth service ticket ttl = 240 # 4m 2023-12-10T18:50:53.912 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-10T18:50:53.912 INFO:teuthology.orchestra.run.smithi033.stdout: # don't complain about insecure global_id in the test suite 2023-12-10T18:50:53.912 INFO:teuthology.orchestra.run.smithi033.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-10T18:50:53.912 INFO:teuthology.orchestra.run.smithi033.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-10T18:50:53.912 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-10T18:50:53.912 INFO:teuthology.orchestra.run.smithi033.stdout: # 1m isn't quite enough 2023-12-10T18:50:53.913 INFO:teuthology.orchestra.run.smithi033.stdout: mon_down_mkfs_grace = 2m 2023-12-10T18:50:53.913 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-10T18:50:53.913 INFO:teuthology.orchestra.run.smithi033.stdout: mon_warn_on_filestore_osds = false 2023-12-10T18:50:53.913 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-10T18:50:53.913 INFO:teuthology.orchestra.run.smithi033.stdout:[client] 2023-12-10T18:50:53.913 INFO:teuthology.orchestra.run.smithi033.stdout: rgw cache enabled = true 2023-12-10T18:50:53.913 INFO:teuthology.orchestra.run.smithi033.stdout: rgw enable ops log = true 2023-12-10T18:50:53.914 INFO:teuthology.orchestra.run.smithi033.stdout: rgw enable usage log = true 2023-12-10T18:50:53.914 INFO:teuthology.orchestra.run.smithi033.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-10T18:50:53.914 INFO:teuthology.orchestra.run.smithi033.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-10T18:50:53.914 INFO:teuthology.orchestra.run.smithi033.stdout: debug rbd = 20 2023-12-10T18:50:53.914 INFO:teuthology.orchestra.run.smithi033.stdout: debug rbd_mirror = 15 2023-12-10T18:50:53.914 INFO:teuthology.orchestra.run.smithi033.stdout: log to stderr = False 2023-12-10T18:50:53.914 INFO:teuthology.orchestra.run.smithi033.stdout: rbd default features = 125 2023-12-10T18:50:53.914 INFO:teuthology.orchestra.run.smithi033.stdout: rbd mirror image policy type = simple 2023-12-10T18:50:53.914 INFO:teuthology.orchestra.run.smithi033.stdout:[mon.a] 2023-12-10T18:50:53.915 INFO:teuthology.orchestra.run.smithi033.stdout:[client.mirror.0] 2023-12-10T18:50:53.915 INFO:teuthology.orchestra.run.smithi033.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-10T18:50:53.915 INFO:teuthology.orchestra.run.smithi033.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-10T18:50:53.915 INFO:teuthology.orchestra.run.smithi033.stdout:[client.mirror.1] 2023-12-10T18:50:53.915 INFO:teuthology.orchestra.run.smithi033.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-10T18:50:53.915 INFO:teuthology.orchestra.run.smithi033.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-10T18:50:53.915 INFO:teuthology.orchestra.run.smithi033.stdout:[client.mirror.2] 2023-12-10T18:50:53.915 INFO:teuthology.orchestra.run.smithi033.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-10T18:50:53.915 INFO:teuthology.orchestra.run.smithi033.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-10T18:50:53.915 INFO:teuthology.orchestra.run.smithi033.stdout:[client.mirror.3] 2023-12-10T18:50:53.915 INFO:teuthology.orchestra.run.smithi033.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-10T18:50:53.915 INFO:teuthology.orchestra.run.smithi033.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-10T18:50:53.922 INFO:teuthology.orchestra.run.smithi076.stdout:[global] 2023-12-10T18:50:53.923 INFO:teuthology.orchestra.run.smithi076.stdout: chdir = "" 2023-12-10T18:50:53.923 INFO:teuthology.orchestra.run.smithi076.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-10T18:50:53.923 INFO:teuthology.orchestra.run.smithi076.stdout: auth supported = cephx 2023-12-10T18:50:53.923 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-10T18:50:53.923 INFO:teuthology.orchestra.run.smithi076.stdout: filestore xattr use omap = true 2023-12-10T18:50:53.923 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-10T18:50:53.923 INFO:teuthology.orchestra.run.smithi076.stdout: mon clock drift allowed = 1.000 2023-12-10T18:50:53.923 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-10T18:50:53.923 INFO:teuthology.orchestra.run.smithi076.stdout: osd crush chooseleaf type = 0 2023-12-10T18:50:53.924 INFO:teuthology.orchestra.run.smithi076.stdout: auth debug = true 2023-12-10T18:50:53.924 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-10T18:50:53.924 INFO:teuthology.orchestra.run.smithi076.stdout: ms die on old message = true 2023-12-10T18:50:53.924 INFO:teuthology.orchestra.run.smithi076.stdout: ms die on bug = true 2023-12-10T18:50:53.924 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-10T18:50:53.924 INFO:teuthology.orchestra.run.smithi076.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-10T18:50:53.924 INFO:teuthology.orchestra.run.smithi076.stdout: mon pg warn max object skew = 0 2023-12-10T18:50:53.924 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-10T18:50:53.924 INFO:teuthology.orchestra.run.smithi076.stdout: # disable pg_autoscaler by default for new pools 2023-12-10T18:50:53.924 INFO:teuthology.orchestra.run.smithi076.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-10T18:50:53.925 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-10T18:50:53.925 INFO:teuthology.orchestra.run.smithi076.stdout: osd pool default size = 2 2023-12-10T18:50:53.925 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-10T18:50:53.925 INFO:teuthology.orchestra.run.smithi076.stdout: mon osd allow primary affinity = true 2023-12-10T18:50:53.925 INFO:teuthology.orchestra.run.smithi076.stdout: mon osd allow pg remap = true 2023-12-10T18:50:53.925 INFO:teuthology.orchestra.run.smithi076.stdout: mon warn on legacy crush tunables = false 2023-12-10T18:50:53.925 INFO:teuthology.orchestra.run.smithi076.stdout: mon warn on crush straw calc version zero = false 2023-12-10T18:50:53.925 INFO:teuthology.orchestra.run.smithi076.stdout: mon warn on no sortbitwise = false 2023-12-10T18:50:53.925 INFO:teuthology.orchestra.run.smithi076.stdout: mon warn on osd down out interval zero = false 2023-12-10T18:50:53.925 INFO:teuthology.orchestra.run.smithi076.stdout: mon warn on too few osds = false 2023-12-10T18:50:53.925 INFO:teuthology.orchestra.run.smithi076.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-10T18:50:53.925 INFO:teuthology.orchestra.run.smithi076.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-10T18:50:53.926 INFO:teuthology.orchestra.run.smithi076.stdout: mon_allow_pool_size_one = true 2023-12-10T18:50:53.926 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-10T18:50:53.926 INFO:teuthology.orchestra.run.smithi076.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-10T18:50:53.926 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-10T18:50:53.926 INFO:teuthology.orchestra.run.smithi076.stdout: osd default data pool replay window = 5 2023-12-10T18:50:53.926 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-10T18:50:53.926 INFO:teuthology.orchestra.run.smithi076.stdout: mon allow pool delete = true 2023-12-10T18:50:53.926 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-10T18:50:53.926 INFO:teuthology.orchestra.run.smithi076.stdout: mon cluster log file level = debug 2023-12-10T18:50:53.927 INFO:teuthology.orchestra.run.smithi076.stdout: debug asserts on shutdown = true 2023-12-10T18:50:53.927 INFO:teuthology.orchestra.run.smithi076.stdout: mon health detail to clog = false 2023-12-10T18:50:53.927 INFO:teuthology.orchestra.run.smithi076.stdout: mon host = 172.21.15.76 2023-12-10T18:50:53.927 INFO:teuthology.orchestra.run.smithi076.stdout: mon client directed command retry = 5 2023-12-10T18:50:53.927 INFO:teuthology.orchestra.run.smithi076.stdout: mon warn on pool no app = False 2023-12-10T18:50:53.927 INFO:teuthology.orchestra.run.smithi076.stdout: ms inject socket failures = 5000 2023-12-10T18:50:53.927 INFO:teuthology.orchestra.run.smithi076.stdout: fsid = af537260-6e4d-44cb-9e7e-e1ba6e0ece8e 2023-12-10T18:50:53.927 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-10T18:50:53.928 INFO:teuthology.orchestra.run.smithi076.stdout:[osd] 2023-12-10T18:50:53.928 INFO:teuthology.orchestra.run.smithi076.stdout: osd journal size = 100 2023-12-10T18:50:53.928 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-10T18:50:53.928 INFO:teuthology.orchestra.run.smithi076.stdout: osd scrub load threshold = 5.0 2023-12-10T18:50:53.928 INFO:teuthology.orchestra.run.smithi076.stdout: osd scrub max interval = 600 2023-12-10T18:50:53.928 INFO:teuthology.orchestra.run.smithi076.stdout: osd mclock profile = high_recovery_ops 2023-12-10T18:50:53.928 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-10T18:50:53.928 INFO:teuthology.orchestra.run.smithi076.stdout: osd recover clone overlap = true 2023-12-10T18:50:53.928 INFO:teuthology.orchestra.run.smithi076.stdout: osd recovery max chunk = 1048576 2023-12-10T18:50:53.928 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-10T18:50:53.928 INFO:teuthology.orchestra.run.smithi076.stdout: osd debug shutdown = true 2023-12-10T18:50:53.929 INFO:teuthology.orchestra.run.smithi076.stdout: osd debug op order = true 2023-12-10T18:50:53.929 INFO:teuthology.orchestra.run.smithi076.stdout: osd debug verify stray on activate = true 2023-12-10T18:50:53.929 INFO:teuthology.orchestra.run.smithi076.stdout: osd debug trim objects = true 2023-12-10T18:50:53.929 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-10T18:50:53.929 INFO:teuthology.orchestra.run.smithi076.stdout: osd open classes on start = true 2023-12-10T18:50:53.929 INFO:teuthology.orchestra.run.smithi076.stdout: osd debug pg log writeout = true 2023-12-10T18:50:53.929 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-10T18:50:53.929 INFO:teuthology.orchestra.run.smithi076.stdout: osd deep scrub update digest min age = 30 2023-12-10T18:50:53.929 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-10T18:50:53.929 INFO:teuthology.orchestra.run.smithi076.stdout: osd map max advance = 10 2023-12-10T18:50:53.930 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-10T18:50:53.930 INFO:teuthology.orchestra.run.smithi076.stdout: journal zero on create = true 2023-12-10T18:50:53.930 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-10T18:50:53.930 INFO:teuthology.orchestra.run.smithi076.stdout: filestore ondisk finisher threads = 3 2023-12-10T18:50:53.930 INFO:teuthology.orchestra.run.smithi076.stdout: filestore apply finisher threads = 3 2023-12-10T18:50:53.930 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-10T18:50:53.930 INFO:teuthology.orchestra.run.smithi076.stdout: bdev debug aio = true 2023-12-10T18:50:53.931 INFO:teuthology.orchestra.run.smithi076.stdout: osd debug misdirected ops = true 2023-12-10T18:50:53.931 INFO:teuthology.orchestra.run.smithi076.stdout: bdev async discard = True 2023-12-10T18:50:53.931 INFO:teuthology.orchestra.run.smithi076.stdout: bdev enable discard = True 2023-12-10T18:50:53.931 INFO:teuthology.orchestra.run.smithi076.stdout: bluestore allocator = stupid 2023-12-10T18:50:53.931 INFO:teuthology.orchestra.run.smithi076.stdout: bluestore block size = 96636764160 2023-12-10T18:50:53.931 INFO:teuthology.orchestra.run.smithi076.stdout: bluestore fsck on mount = True 2023-12-10T18:50:53.932 INFO:teuthology.orchestra.run.smithi076.stdout: debug bluefs = 1/20 2023-12-10T18:50:53.932 INFO:teuthology.orchestra.run.smithi076.stdout: debug bluestore = 1/20 2023-12-10T18:50:53.932 INFO:teuthology.orchestra.run.smithi076.stdout: debug ms = 1 2023-12-10T18:50:53.932 INFO:teuthology.orchestra.run.smithi076.stdout: debug osd = 20 2023-12-10T18:50:53.932 INFO:teuthology.orchestra.run.smithi076.stdout: debug rocksdb = 4/10 2023-12-10T18:50:53.933 INFO:teuthology.orchestra.run.smithi076.stdout: mon osd backfillfull_ratio = 0.85 2023-12-10T18:50:53.933 INFO:teuthology.orchestra.run.smithi076.stdout: mon osd full ratio = 0.9 2023-12-10T18:50:53.933 INFO:teuthology.orchestra.run.smithi076.stdout: mon osd nearfull ratio = 0.8 2023-12-10T18:50:53.933 INFO:teuthology.orchestra.run.smithi076.stdout: osd failsafe full ratio = 0.95 2023-12-10T18:50:53.933 INFO:teuthology.orchestra.run.smithi076.stdout: osd objectstore = bluestore 2023-12-10T18:50:53.933 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-10T18:50:53.933 INFO:teuthology.orchestra.run.smithi076.stdout:[mgr] 2023-12-10T18:50:53.933 INFO:teuthology.orchestra.run.smithi076.stdout: debug ms = 1 2023-12-10T18:50:53.934 INFO:teuthology.orchestra.run.smithi076.stdout: debug mgr = 20 2023-12-10T18:50:53.934 INFO:teuthology.orchestra.run.smithi076.stdout: debug mon = 20 2023-12-10T18:50:53.934 INFO:teuthology.orchestra.run.smithi076.stdout: debug auth = 20 2023-12-10T18:50:53.934 INFO:teuthology.orchestra.run.smithi076.stdout: mon reweight min pgs per osd = 4 2023-12-10T18:50:53.934 INFO:teuthology.orchestra.run.smithi076.stdout: mon reweight min bytes per osd = 10 2023-12-10T18:50:53.934 INFO:teuthology.orchestra.run.smithi076.stdout: mgr/telemetry/nag = false 2023-12-10T18:50:53.934 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-10T18:50:53.935 INFO:teuthology.orchestra.run.smithi076.stdout:[mon] 2023-12-10T18:50:53.935 INFO:teuthology.orchestra.run.smithi076.stdout: debug ms = 1 2023-12-10T18:50:53.935 INFO:teuthology.orchestra.run.smithi076.stdout: debug mon = 20 2023-12-10T18:50:53.935 INFO:teuthology.orchestra.run.smithi076.stdout: debug paxos = 20 2023-12-10T18:50:53.935 INFO:teuthology.orchestra.run.smithi076.stdout: debug auth = 20 2023-12-10T18:50:53.935 INFO:teuthology.orchestra.run.smithi076.stdout: mon data avail warn = 5 2023-12-10T18:50:53.935 INFO:teuthology.orchestra.run.smithi076.stdout: mon mgr mkfs grace = 240 2023-12-10T18:50:53.936 INFO:teuthology.orchestra.run.smithi076.stdout: mon reweight min pgs per osd = 4 2023-12-10T18:50:53.936 INFO:teuthology.orchestra.run.smithi076.stdout: mon osd reporter subtree level = osd 2023-12-10T18:50:53.936 INFO:teuthology.orchestra.run.smithi076.stdout: mon osd prime pg temp = true 2023-12-10T18:50:53.936 INFO:teuthology.orchestra.run.smithi076.stdout: mon reweight min bytes per osd = 10 2023-12-10T18:50:53.936 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-10T18:50:53.936 INFO:teuthology.orchestra.run.smithi076.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-10T18:50:53.936 INFO:teuthology.orchestra.run.smithi076.stdout: auth mon ticket ttl = 660 # 11m 2023-12-10T18:50:53.937 INFO:teuthology.orchestra.run.smithi076.stdout: auth service ticket ttl = 240 # 4m 2023-12-10T18:50:53.937 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-10T18:50:53.937 INFO:teuthology.orchestra.run.smithi076.stdout: # don't complain about insecure global_id in the test suite 2023-12-10T18:50:53.937 INFO:teuthology.orchestra.run.smithi076.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-10T18:50:53.937 INFO:teuthology.orchestra.run.smithi076.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-10T18:50:53.937 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-10T18:50:53.938 INFO:teuthology.orchestra.run.smithi076.stdout: # 1m isn't quite enough 2023-12-10T18:50:53.938 INFO:teuthology.orchestra.run.smithi076.stdout: mon_down_mkfs_grace = 2m 2023-12-10T18:50:53.938 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-10T18:50:53.938 INFO:teuthology.orchestra.run.smithi076.stdout: mon_warn_on_filestore_osds = false 2023-12-10T18:50:53.938 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-10T18:50:53.938 INFO:teuthology.orchestra.run.smithi076.stdout:[client] 2023-12-10T18:50:53.938 INFO:teuthology.orchestra.run.smithi076.stdout: rgw cache enabled = true 2023-12-10T18:50:53.939 INFO:teuthology.orchestra.run.smithi076.stdout: rgw enable ops log = true 2023-12-10T18:50:53.939 INFO:teuthology.orchestra.run.smithi076.stdout: rgw enable usage log = true 2023-12-10T18:50:53.939 INFO:teuthology.orchestra.run.smithi076.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-10T18:50:53.939 INFO:teuthology.orchestra.run.smithi076.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-10T18:50:53.939 INFO:teuthology.orchestra.run.smithi076.stdout: debug rbd = 20 2023-12-10T18:50:53.939 INFO:teuthology.orchestra.run.smithi076.stdout: debug rbd_mirror = 15 2023-12-10T18:50:53.939 INFO:teuthology.orchestra.run.smithi076.stdout: log to stderr = False 2023-12-10T18:50:53.939 INFO:teuthology.orchestra.run.smithi076.stdout: rbd default features = 125 2023-12-10T18:50:53.939 INFO:teuthology.orchestra.run.smithi076.stdout: rbd mirror image policy type = simple 2023-12-10T18:50:53.940 INFO:teuthology.orchestra.run.smithi076.stdout:[mon.a] 2023-12-10T18:50:53.940 INFO:teuthology.orchestra.run.smithi076.stdout:[client.mirror.0] 2023-12-10T18:50:53.940 INFO:teuthology.orchestra.run.smithi076.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-10T18:50:53.940 INFO:teuthology.orchestra.run.smithi076.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-10T18:50:53.940 INFO:teuthology.orchestra.run.smithi076.stdout:[client.mirror.1] 2023-12-10T18:50:53.940 INFO:teuthology.orchestra.run.smithi076.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-10T18:50:53.940 INFO:teuthology.orchestra.run.smithi076.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-10T18:50:53.940 INFO:teuthology.orchestra.run.smithi076.stdout:[client.mirror.2] 2023-12-10T18:50:53.940 INFO:teuthology.orchestra.run.smithi076.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-10T18:50:53.940 INFO:teuthology.orchestra.run.smithi076.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-10T18:50:53.941 INFO:teuthology.orchestra.run.smithi076.stdout:[client.mirror.3] 2023-12-10T18:50:53.941 INFO:teuthology.orchestra.run.smithi076.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-10T18:50:53.941 INFO:teuthology.orchestra.run.smithi076.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-10T18:50:53.942 INFO:tasks.ceph:Creating admin key on cluster2.mon.a... 2023-12-10T18:50:53.942 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster2.keyring 2023-12-10T18:50:54.034 INFO:tasks.ceph:Copying monmap to all nodes... 2023-12-10T18:50:54.034 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2023-12-10T18:50:54.034 DEBUG:teuthology.orchestra.run.smithi076:> dd if=/etc/ceph/cluster2.keyring of=/dev/stdout 2023-12-10T18:50:54.040 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2023-12-10T18:50:54.040 DEBUG:teuthology.orchestra.run.smithi076:> dd if=/home/ubuntu/cephtest/cluster2.monmap of=/dev/stdout 2023-12-10T18:50:54.085 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi033.front.sepia.ceph.com 2023-12-10T18:50:54.086 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-10T18:50:54.086 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/ceph/cluster2.keyring 2023-12-10T18:50:54.086 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2023-12-10T18:50:54.111 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-10T18:50:54.111 DEBUG:teuthology.orchestra.run.smithi033:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2023-12-10T18:50:54.163 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi076.front.sepia.ceph.com 2023-12-10T18:50:54.164 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2023-12-10T18:50:54.164 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/etc/ceph/cluster2.keyring 2023-12-10T18:50:54.164 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2023-12-10T18:50:54.186 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2023-12-10T18:50:54.186 DEBUG:teuthology.orchestra.run.smithi076:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2023-12-10T18:50:54.234 INFO:tasks.ceph:Setting up mon nodes... 2023-12-10T18:50:54.235 INFO:tasks.ceph:Setting up mgr nodes... 2023-12-10T18:50:54.235 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /var/lib/ceph/mgr/cluster2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster2-x/keyring 2023-12-10T18:50:54.282 INFO:teuthology.orchestra.run.smithi033.stdout:creating /var/lib/ceph/mgr/cluster2-x/keyring 2023-12-10T18:50:54.285 INFO:tasks.ceph:Setting up mds nodes... 2023-12-10T18:50:54.285 INFO:tasks.ceph_client:Setting up client nodes... 2023-12-10T18:50:54.285 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster2.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.0.keyring 2023-12-10T18:50:54.360 INFO:teuthology.orchestra.run.smithi033.stdout:creating /etc/ceph/cluster2.client.0.keyring 2023-12-10T18:50:54.371 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster2.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.keyring 2023-12-10T18:50:54.408 INFO:teuthology.orchestra.run.smithi076.stdout:creating /etc/ceph/cluster2.client.mirror.keyring 2023-12-10T18:50:54.419 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster2.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.0.keyring 2023-12-10T18:50:54.492 INFO:teuthology.orchestra.run.smithi076.stdout:creating /etc/ceph/cluster2.client.mirror.0.keyring 2023-12-10T18:50:54.502 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster2.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.1.keyring 2023-12-10T18:50:54.573 INFO:teuthology.orchestra.run.smithi076.stdout:creating /etc/ceph/cluster2.client.mirror.1.keyring 2023-12-10T18:50:54.583 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster2.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.2.keyring 2023-12-10T18:50:54.657 INFO:teuthology.orchestra.run.smithi076.stdout:creating /etc/ceph/cluster2.client.mirror.2.keyring 2023-12-10T18:50:54.668 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster2.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.3.keyring 2023-12-10T18:50:54.743 INFO:teuthology.orchestra.run.smithi076.stdout:creating /etc/ceph/cluster2.client.mirror.3.keyring 2023-12-10T18:50:54.752 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster2.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.4.keyring 2023-12-10T18:50:54.826 INFO:teuthology.orchestra.run.smithi076.stdout:creating /etc/ceph/cluster2.client.mirror.4.keyring 2023-12-10T18:50:54.836 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster2.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.5.keyring 2023-12-10T18:50:54.909 INFO:teuthology.orchestra.run.smithi076.stdout:creating /etc/ceph/cluster2.client.mirror.5.keyring 2023-12-10T18:50:54.919 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster2.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.6.keyring 2023-12-10T18:50:54.991 INFO:teuthology.orchestra.run.smithi076.stdout:creating /etc/ceph/cluster2.client.mirror.6.keyring 2023-12-10T18:50:55.001 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-12-10T18:50:55.001 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi033.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi076.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2023-12-10T18:50:55.001 DEBUG:teuthology.orchestra.run.smithi076:> sudo mkdir -p /var/lib/ceph/osd/cluster2-0 2023-12-10T18:50:55.061 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-10T18:50:55.062 INFO:tasks.ceph:role: cluster2.osd.0 2023-12-10T18:50:55.062 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi076.front.sepia.ceph.com 2023-12-10T18:50:55.062 DEBUG:teuthology.orchestra.run.smithi076:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-12-10T18:50:55.125 INFO:teuthology.orchestra.run.smithi076.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-12-10T18:50:55.126 INFO:teuthology.orchestra.run.smithi076.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-10T18:50:55.126 INFO:teuthology.orchestra.run.smithi076.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-10T18:50:55.126 INFO:teuthology.orchestra.run.smithi076.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-12-10T18:50:55.126 INFO:teuthology.orchestra.run.smithi076.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-10T18:50:55.126 INFO:teuthology.orchestra.run.smithi076.stdout: = sunit=0 swidth=0 blks 2023-12-10T18:50:55.126 INFO:teuthology.orchestra.run.smithi076.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-10T18:50:55.126 INFO:teuthology.orchestra.run.smithi076.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-10T18:50:55.126 INFO:teuthology.orchestra.run.smithi076.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-10T18:50:55.126 INFO:teuthology.orchestra.run.smithi076.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-10T18:50:55.586 INFO:teuthology.orchestra.run.smithi076.stdout:Discarding blocks...Done. 2023-12-10T18:50:55.588 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi076.front.sepia.ceph.com -o noatime 2023-12-10T18:50:55.589 DEBUG:teuthology.orchestra.run.smithi076:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster2-0 2023-12-10T18:50:55.678 DEBUG:teuthology.orchestra.run.smithi076:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-0 2023-12-10T18:50:55.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-10T18:50:55.690 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: /sbin/restorecon: command not found 2023-12-10T18:50:55.690 DEBUG:teuthology.orchestra.run.smithi076:> sudo mkdir -p /var/lib/ceph/osd/cluster2-1 2023-12-10T18:50:55.745 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-10T18:50:55.746 INFO:tasks.ceph:role: cluster2.osd.1 2023-12-10T18:50:55.746 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi076.front.sepia.ceph.com 2023-12-10T18:50:55.746 DEBUG:teuthology.orchestra.run.smithi076:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-12-10T18:50:55.810 INFO:teuthology.orchestra.run.smithi076.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-12-10T18:50:55.810 INFO:teuthology.orchestra.run.smithi076.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-10T18:50:55.810 INFO:teuthology.orchestra.run.smithi076.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-10T18:50:55.810 INFO:teuthology.orchestra.run.smithi076.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-12-10T18:50:55.810 INFO:teuthology.orchestra.run.smithi076.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-10T18:50:55.810 INFO:teuthology.orchestra.run.smithi076.stdout: = sunit=0 swidth=0 blks 2023-12-10T18:50:55.811 INFO:teuthology.orchestra.run.smithi076.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-10T18:50:55.811 INFO:teuthology.orchestra.run.smithi076.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-10T18:50:55.811 INFO:teuthology.orchestra.run.smithi076.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-10T18:50:55.811 INFO:teuthology.orchestra.run.smithi076.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-10T18:50:56.281 INFO:teuthology.orchestra.run.smithi076.stdout:Discarding blocks...Done. 2023-12-10T18:50:56.283 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi076.front.sepia.ceph.com -o noatime 2023-12-10T18:50:56.283 DEBUG:teuthology.orchestra.run.smithi076:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster2-1 2023-12-10T18:50:56.434 DEBUG:teuthology.orchestra.run.smithi076:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-1 2023-12-10T18:50:56.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-10T18:50:56.446 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: /sbin/restorecon: command not found 2023-12-10T18:50:56.446 DEBUG:teuthology.orchestra.run.smithi076:> sudo mkdir -p /var/lib/ceph/osd/cluster2-2 2023-12-10T18:50:56.502 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-10T18:50:56.502 INFO:tasks.ceph:role: cluster2.osd.2 2023-12-10T18:50:56.503 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi076.front.sepia.ceph.com 2023-12-10T18:50:56.503 DEBUG:teuthology.orchestra.run.smithi076:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-12-10T18:50:56.566 INFO:teuthology.orchestra.run.smithi076.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-12-10T18:50:56.566 INFO:teuthology.orchestra.run.smithi076.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-10T18:50:56.566 INFO:teuthology.orchestra.run.smithi076.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-10T18:50:56.566 INFO:teuthology.orchestra.run.smithi076.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-12-10T18:50:56.566 INFO:teuthology.orchestra.run.smithi076.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-10T18:50:56.566 INFO:teuthology.orchestra.run.smithi076.stdout: = sunit=0 swidth=0 blks 2023-12-10T18:50:56.566 INFO:teuthology.orchestra.run.smithi076.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-10T18:50:56.566 INFO:teuthology.orchestra.run.smithi076.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-10T18:50:56.567 INFO:teuthology.orchestra.run.smithi076.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-10T18:50:56.567 INFO:teuthology.orchestra.run.smithi076.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-10T18:50:57.018 INFO:teuthology.orchestra.run.smithi076.stdout:Discarding blocks...Done. 2023-12-10T18:50:57.020 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi076.front.sepia.ceph.com -o noatime 2023-12-10T18:50:57.021 DEBUG:teuthology.orchestra.run.smithi076:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster2-2 2023-12-10T18:50:57.150 DEBUG:teuthology.orchestra.run.smithi076:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-2 2023-12-10T18:50:57.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-10T18:50:57.162 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: /sbin/restorecon: command not found 2023-12-10T18:50:57.162 DEBUG:teuthology.orchestra.run.smithi076:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster2.monmap 2023-12-10T18:50:57.282 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-10T18:50:57.278+0000 7f146690d640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-0/keyring: can't open /var/lib/ceph/osd/cluster2-0/keyring: (2) No such file or directory 2023-12-10T18:50:57.286 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-10T18:50:57.282+0000 7f146690d640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-0/keyring 2023-12-10T18:50:57.286 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-10T18:50:57.282+0000 7f146690d640 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2023-12-10T18:50:57.286 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-10T18:50:57.282+0000 7f146690d640 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2023-12-10T18:50:57.286 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-10T18:50:57.282+0000 7f146690d640 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2023-12-10T18:50:57.287 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-10T18:50:57.282+0000 7f146690d640 -1 bluestore(/var/lib/ceph/osd/cluster2-0) _read_fsid unparsable uuid 2023-12-10T18:51:00.291 DEBUG:teuthology.orchestra.run.smithi076:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-0 2023-12-10T18:51:00.305 DEBUG:teuthology.orchestra.run.smithi076:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster2.monmap 2023-12-10T18:51:00.393 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-10T18:51:00.386+0000 7feea9f1b640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-1/keyring: can't open /var/lib/ceph/osd/cluster2-1/keyring: (2) No such file or directory 2023-12-10T18:51:00.393 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-10T18:51:00.390+0000 7feea9f1b640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-1/keyring 2023-12-10T18:51:00.393 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-10T18:51:00.390+0000 7feea9f1b640 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2023-12-10T18:51:00.393 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-10T18:51:00.390+0000 7feea9f1b640 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2023-12-10T18:51:00.393 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-10T18:51:00.390+0000 7feea9f1b640 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2023-12-10T18:51:00.394 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-10T18:51:00.390+0000 7feea9f1b640 -1 bluestore(/var/lib/ceph/osd/cluster2-1) _read_fsid unparsable uuid 2023-12-10T18:51:03.430 DEBUG:teuthology.orchestra.run.smithi076:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-1 2023-12-10T18:51:03.486 DEBUG:teuthology.orchestra.run.smithi076:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster2.monmap 2023-12-10T18:51:03.569 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-10T18:51:03.566+0000 7f50fe6bc640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-2/keyring: can't open /var/lib/ceph/osd/cluster2-2/keyring: (2) No such file or directory 2023-12-10T18:51:03.570 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-10T18:51:03.566+0000 7f50fe6bc640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-2/keyring 2023-12-10T18:51:03.570 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-10T18:51:03.566+0000 7f50fe6bc640 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2023-12-10T18:51:03.570 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-10T18:51:03.566+0000 7f50fe6bc640 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2023-12-10T18:51:03.571 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-10T18:51:03.566+0000 7f50fe6bc640 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2023-12-10T18:51:03.571 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-10T18:51:03.566+0000 7f50fe6bc640 -1 bluestore(/var/lib/ceph/osd/cluster2-2) _read_fsid unparsable uuid 2023-12-10T18:51:06.573 DEBUG:teuthology.orchestra.run.smithi076:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-2 2023-12-10T18:51:06.629 INFO:tasks.ceph:Reading keys from all nodes... 2023-12-10T18:51:06.630 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-10T18:51:06.630 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/mgr/cluster2-x/keyring of=/dev/stdout 2023-12-10T18:51:06.645 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2023-12-10T18:51:06.645 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/var/lib/ceph/osd/cluster2-0/keyring of=/dev/stdout 2023-12-10T18:51:06.685 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2023-12-10T18:51:06.685 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/var/lib/ceph/osd/cluster2-1/keyring of=/dev/stdout 2023-12-10T18:51:06.740 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2023-12-10T18:51:06.740 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/var/lib/ceph/osd/cluster2-2/keyring of=/dev/stdout 2023-12-10T18:51:06.797 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-10T18:51:06.797 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/etc/ceph/cluster2.client.0.keyring of=/dev/stdout 2023-12-10T18:51:06.804 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2023-12-10T18:51:06.804 DEBUG:teuthology.orchestra.run.smithi076:> dd if=/etc/ceph/cluster2.client.mirror.keyring of=/dev/stdout 2023-12-10T18:51:06.846 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2023-12-10T18:51:06.846 DEBUG:teuthology.orchestra.run.smithi076:> dd if=/etc/ceph/cluster2.client.mirror.0.keyring of=/dev/stdout 2023-12-10T18:51:06.897 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2023-12-10T18:51:06.898 DEBUG:teuthology.orchestra.run.smithi076:> dd if=/etc/ceph/cluster2.client.mirror.1.keyring of=/dev/stdout 2023-12-10T18:51:06.945 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2023-12-10T18:51:06.945 DEBUG:teuthology.orchestra.run.smithi076:> dd if=/etc/ceph/cluster2.client.mirror.2.keyring of=/dev/stdout 2023-12-10T18:51:06.993 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2023-12-10T18:51:06.993 DEBUG:teuthology.orchestra.run.smithi076:> dd if=/etc/ceph/cluster2.client.mirror.3.keyring of=/dev/stdout 2023-12-10T18:51:07.041 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2023-12-10T18:51:07.041 DEBUG:teuthology.orchestra.run.smithi076:> dd if=/etc/ceph/cluster2.client.mirror.4.keyring of=/dev/stdout 2023-12-10T18:51:07.089 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2023-12-10T18:51:07.090 DEBUG:teuthology.orchestra.run.smithi076:> dd if=/etc/ceph/cluster2.client.mirror.5.keyring of=/dev/stdout 2023-12-10T18:51:07.137 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2023-12-10T18:51:07.137 DEBUG:teuthology.orchestra.run.smithi076:> dd if=/etc/ceph/cluster2.client.mirror.6.keyring of=/dev/stdout 2023-12-10T18:51:07.185 INFO:tasks.ceph:Adding keys to all mons... 2023-12-10T18:51:07.186 DEBUG:teuthology.orchestra.run.smithi076:> sudo tee -a /etc/ceph/cluster2.keyring 2023-12-10T18:51:07.240 INFO:teuthology.orchestra.run.smithi076.stdout:[mgr.x] 2023-12-10T18:51:07.240 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQCOCHZl20HdEBAA/e2qvYbv8Kzr57i5HVCvqg== 2023-12-10T18:51:07.240 INFO:teuthology.orchestra.run.smithi076.stdout:[osd.0] 2023-12-10T18:51:07.240 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQCRCHZlEly5EBAAQ7HOGgXpcZXVHc2pqfqpBw== 2023-12-10T18:51:07.240 INFO:teuthology.orchestra.run.smithi076.stdout:[osd.1] 2023-12-10T18:51:07.240 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQCUCHZlhp5UFxAAIAEWEUrH/xBC8zKZ3cwmmA== 2023-12-10T18:51:07.240 INFO:teuthology.orchestra.run.smithi076.stdout:[osd.2] 2023-12-10T18:51:07.240 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQCXCHZl/7LfIRAAMeNbAREdSYjFnxLlWFdmow== 2023-12-10T18:51:07.240 INFO:teuthology.orchestra.run.smithi076.stdout:[client.0] 2023-12-10T18:51:07.240 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQCOCHZlxL2DFRAA6XD2lAE0xvMLygvDUbqTfg== 2023-12-10T18:51:07.241 INFO:teuthology.orchestra.run.smithi076.stdout:[client.mirror] 2023-12-10T18:51:07.241 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQCOCHZl5lM6GBAAy2ylKP0J9DtNfzabMmPHuA== 2023-12-10T18:51:07.241 INFO:teuthology.orchestra.run.smithi076.stdout:[client.mirror.0] 2023-12-10T18:51:07.241 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQCOCHZl8oU+HRAA+QwAThxlqSY6m3L1JmJP1Q== 2023-12-10T18:51:07.241 INFO:teuthology.orchestra.run.smithi076.stdout:[client.mirror.1] 2023-12-10T18:51:07.241 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQCOCHZljsMXIhAA8P4oBaTgS6HWZ7xK00SEGA== 2023-12-10T18:51:07.241 INFO:teuthology.orchestra.run.smithi076.stdout:[client.mirror.2] 2023-12-10T18:51:07.241 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQCOCHZlKZEXJxAA6PcTShEf7TZr/qupREEM0w== 2023-12-10T18:51:07.241 INFO:teuthology.orchestra.run.smithi076.stdout:[client.mirror.3] 2023-12-10T18:51:07.241 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQCOCHZlIAQvLBAAZaPD6DfyCBBgDCL86s4irA== 2023-12-10T18:51:07.241 INFO:teuthology.orchestra.run.smithi076.stdout:[client.mirror.4] 2023-12-10T18:51:07.241 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQCOCHZlQMAvMRAAfGt1HMJg5ukEsPtjo0Vo2g== 2023-12-10T18:51:07.241 INFO:teuthology.orchestra.run.smithi076.stdout:[client.mirror.5] 2023-12-10T18:51:07.241 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQCOCHZlybsXNhAAiQTKhLPd/dQEtlJz8FDUEA== 2023-12-10T18:51:07.241 INFO:teuthology.orchestra.run.smithi076.stdout:[client.mirror.6] 2023-12-10T18:51:07.242 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQCOCHZl6Uv6OhAA315cDLVUmcJpufs5qNrL/w== 2023-12-10T18:51:07.242 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-10T18:51:07.330 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-10T18:51:07.374 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-10T18:51:07.454 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-10T18:51:07.537 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-10T18:51:07.628 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-10T18:51:07.670 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-10T18:51:07.754 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-10T18:51:07.843 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-10T18:51:07.886 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-10T18:51:07.971 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-10T18:51:08.060 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-10T18:51:08.102 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-10T18:51:08.186 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-12-10T18:51:08.187 DEBUG:teuthology.orchestra.run.smithi076:> sudo mkdir -p /var/lib/ceph/mon/cluster2-a 2023-12-10T18:51:08.240 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster2 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster2.monmap --keyring /etc/ceph/cluster2.keyring 2023-12-10T18:51:08.336 DEBUG:teuthology.orchestra.run.smithi076:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster2-a 2023-12-10T18:51:08.347 DEBUG:teuthology.orchestra.run.smithi076:> rm -- /home/ubuntu/cephtest/cluster2.monmap 2023-12-10T18:51:08.392 INFO:tasks.ceph:Starting mon daemons in cluster cluster2... 2023-12-10T18:51:08.392 INFO:tasks.ceph.cluster2.mon.a:Restarting daemon 2023-12-10T18:51:08.392 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster2 -i a 2023-12-10T18:51:08.434 INFO:tasks.ceph.cluster2.mon.a:Started 2023-12-10T18:51:08.435 INFO:tasks.ceph:Starting mgr daemons in cluster cluster2... 2023-12-10T18:51:08.435 INFO:tasks.ceph.cluster2.mgr.x:Restarting daemon 2023-12-10T18:51:08.435 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster2 -i x 2023-12-10T18:51:08.438 INFO:tasks.ceph.cluster2.mgr.x:Started 2023-12-10T18:51:08.438 INFO:tasks.ceph:Setting crush tunables to default 2023-12-10T18:51:08.439 DEBUG:teuthology.orchestra.run.smithi076:> sudo ceph --cluster cluster2 osd crush tunables default 2023-12-10T18:51:08.669 INFO:teuthology.orchestra.run.smithi076.stderr:adjusted tunables profile to default 2023-12-10T18:51:08.687 INFO:tasks.ceph:check_enable_crimson: False 2023-12-10T18:51:08.687 INFO:tasks.ceph:Starting osd daemons in cluster cluster2... 2023-12-10T18:51:08.688 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2023-12-10T18:51:08.688 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/var/lib/ceph/osd/cluster2-0/fsid of=/dev/stdout 2023-12-10T18:51:08.698 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2023-12-10T18:51:08.698 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/var/lib/ceph/osd/cluster2-1/fsid of=/dev/stdout 2023-12-10T18:51:08.707 INFO:tasks.ceph.cluster2.mgr.x.smithi033.stderr:2023-12-10T18:51:08.703+0000 7ff86b5de180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-10T18:51:08.751 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2023-12-10T18:51:08.751 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/var/lib/ceph/osd/cluster2-2/fsid of=/dev/stdout 2023-12-10T18:51:08.804 DEBUG:teuthology.orchestra.run.smithi076:> sudo ceph --cluster cluster2 osd new f61274b1-3e55-4802-a1da-3728a61b0bf5 0 2023-12-10T18:51:09.047 INFO:teuthology.orchestra.run.smithi076.stdout:0 2023-12-10T18:51:09.065 DEBUG:teuthology.orchestra.run.smithi076:> sudo ceph --cluster cluster2 osd new 7f08d0f1-6812-485c-9efa-0741603162e4 1 2023-12-10T18:51:09.112 INFO:tasks.ceph.cluster2.mgr.x.smithi033.stderr:2023-12-10T18:51:09.107+0000 7ff86b5de180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-10T18:51:09.255 INFO:tasks.ceph.cluster2.mgr.x.smithi033.stderr:2023-12-10T18:51:09.251+0000 7ff86b5de180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-10T18:51:09.258 INFO:teuthology.orchestra.run.smithi076.stdout:1 2023-12-10T18:51:09.275 DEBUG:teuthology.orchestra.run.smithi076:> sudo ceph --cluster cluster2 osd new 8fd7962b-5418-4161-98a5-16a26bf0e050 2 2023-12-10T18:51:09.461 INFO:teuthology.orchestra.run.smithi076.stdout:2 2023-12-10T18:51:09.478 INFO:tasks.ceph.cluster2.osd.0:Restarting daemon 2023-12-10T18:51:09.478 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0 2023-12-10T18:51:09.481 INFO:tasks.ceph.cluster2.osd.0:Started 2023-12-10T18:51:09.481 INFO:tasks.ceph.cluster2.osd.1:Restarting daemon 2023-12-10T18:51:09.481 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 2023-12-10T18:51:09.483 INFO:tasks.ceph.cluster2.osd.1:Started 2023-12-10T18:51:09.483 INFO:tasks.ceph.cluster2.osd.2:Restarting daemon 2023-12-10T18:51:09.483 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 2023-12-10T18:51:09.485 INFO:tasks.ceph.cluster2.osd.2:Started 2023-12-10T18:51:09.486 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2023-12-10T18:51:09.664 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-10T18:51:09.664 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":5,"fsid":"af537260-6e4d-44cb-9e7e-e1ba6e0ece8e","created":"2023-12-10T18:51:08.563100+0000","modified":"2023-12-10T18:51:09.458539+0000","last_up_change":"0.000000","last_in_change":"2023-12-10T18:51:09.458539+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f61274b1-3e55-4802-a1da-3728a61b0bf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"7f08d0f1-6812-485c-9efa-0741603162e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8fd7962b-5418-4161-98a5-16a26bf0e050","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-10T18:51:09.681 INFO:tasks.ceph.ceph_manager.cluster2:[] 2023-12-10T18:51:09.681 INFO:tasks.ceph:Waiting for OSDs to come up 2023-12-10T18:51:09.697 INFO:tasks.ceph.cluster2.mgr.x.smithi033.stderr:2023-12-10T18:51:09.691+0000 7ff86b5de180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-10T18:51:09.750 INFO:tasks.ceph.cluster2.mgr.x.smithi033.stderr:2023-12-10T18:51:09.743+0000 7ff86b5de180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-10T18:51:09.800 INFO:tasks.ceph.cluster2.mgr.x.smithi033.stderr:2023-12-10T18:51:09.795+0000 7ff86b5de180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-10T18:51:09.876 INFO:tasks.ceph.cluster2.mgr.x.smithi033.stderr:2023-12-10T18:51:09.871+0000 7ff86b5de180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-10T18:51:09.928 INFO:tasks.ceph.cluster2.mgr.x.smithi033.stderr:2023-12-10T18:51:09.923+0000 7ff86b5de180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-10T18:51:09.983 DEBUG:teuthology.orchestra.run.smithi076:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2023-12-10T18:51:10.004 INFO:tasks.ceph.cluster2.mgr.x.smithi033.stderr:2023-12-10T18:51:09.999+0000 7ff86b5de180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-10T18:51:10.054 INFO:tasks.ceph.cluster2.mgr.x.smithi033.stderr:2023-12-10T18:51:10.051+0000 7ff86b5de180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-10T18:51:10.094 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr:2023-12-10T18:51:10.090+0000 7fa383c30640 -1 Falling back to public interface 2023-12-10T18:51:10.094 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr:2023-12-10T18:51:10.090+0000 7ff7b8684640 -1 Falling back to public interface 2023-12-10T18:51:10.095 INFO:tasks.ceph.cluster2.osd.2.smithi076.stderr:2023-12-10T18:51:10.090+0000 7f7f7c3b9640 -1 Falling back to public interface 2023-12-10T18:51:10.168 INFO:teuthology.misc.health.smithi076.stdout: 2023-12-10T18:51:10.168 INFO:teuthology.misc.health.smithi076.stdout:{"epoch":5,"fsid":"af537260-6e4d-44cb-9e7e-e1ba6e0ece8e","created":"2023-12-10T18:51:08.563100+0000","modified":"2023-12-10T18:51:09.458539+0000","last_up_change":"0.000000","last_in_change":"2023-12-10T18:51:09.458539+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f61274b1-3e55-4802-a1da-3728a61b0bf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"7f08d0f1-6812-485c-9efa-0741603162e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8fd7962b-5418-4161-98a5-16a26bf0e050","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-10T18:51:10.182 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-12-10T18:51:10.188 INFO:tasks.ceph.cluster2.mgr.x.smithi033.stderr:2023-12-10T18:51:10.183+0000 7ff86b5de180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-10T18:51:10.241 INFO:tasks.ceph.cluster2.mgr.x.smithi033.stderr:2023-12-10T18:51:10.235+0000 7ff86b5de180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-10T18:51:10.464 INFO:tasks.ceph.cluster2.mgr.x.smithi033.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-12-10T18:51:10.464 INFO:tasks.ceph.cluster2.mgr.x.smithi033.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-12-10T18:51:10.464 INFO:tasks.ceph.cluster2.mgr.x.smithi033.stderr: from numpy import show_config as show_numpy_config 2023-12-10T18:51:10.471 INFO:tasks.ceph.cluster2.mgr.x.smithi033.stderr:2023-12-10T18:51:10.467+0000 7ff86b5de180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-12-10T18:51:10.598 INFO:tasks.ceph.cluster2.mgr.x.smithi033.stderr:2023-12-10T18:51:10.591+0000 7ff86b5de180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-10T18:51:10.711 INFO:tasks.ceph.cluster2.mgr.x.smithi033.stderr:2023-12-10T18:51:10.707+0000 7ff86b5de180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-10T18:51:10.774 INFO:tasks.ceph.cluster2.mgr.x.smithi033.stderr:2023-12-10T18:51:10.767+0000 7ff86b5de180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-10T18:51:10.901 INFO:tasks.ceph.cluster2.mgr.x.smithi033.stderr:2023-12-10T18:51:10.895+0000 7ff86b5de180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-10T18:51:11.392 INFO:tasks.ceph.cluster2.mgr.x.smithi033.stderr:2023-12-10T18:51:11.387+0000 7ff86b5de180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-10T18:51:11.468 INFO:tasks.ceph.cluster2.mgr.x.smithi033.stderr:2023-12-10T18:51:11.463+0000 7ff86b5de180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-10T18:51:11.606 INFO:tasks.ceph.cluster2.mgr.x.smithi033.stderr:2023-12-10T18:51:11.603+0000 7ff86b5de180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-10T18:51:11.681 INFO:tasks.ceph.cluster2.mgr.x.smithi033.stderr:2023-12-10T18:51:11.675+0000 7ff86b5de180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-10T18:51:11.973 INFO:tasks.ceph.cluster2.mgr.x.smithi033.stderr:2023-12-10T18:51:11.967+0000 7ff86b5de180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-10T18:51:12.027 INFO:tasks.ceph.cluster2.mgr.x.smithi033.stderr:2023-12-10T18:51:12.023+0000 7ff86b5de180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-10T18:51:13.041 INFO:tasks.ceph.cluster2.mgr.x.smithi033.stderr:2023-12-10T18:51:13.035+0000 7ff812d29640 -1 mgr.server handle_report got status from non-daemon mon.a 2023-12-10T18:51:13.853 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr:2023-12-10T18:51:13.850+0000 7fa383c30640 -1 osd.1 0 log_to_monitors true 2023-12-10T18:51:13.973 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr:2023-12-10T18:51:13.966+0000 7ff7b8684640 -1 osd.0 0 log_to_monitors true 2023-12-10T18:51:13.979 INFO:tasks.ceph.cluster2.osd.2.smithi076.stderr:2023-12-10T18:51:13.974+0000 7f7f7c3b9640 -1 osd.2 0 log_to_monitors true 2023-12-10T18:51:16.486 DEBUG:teuthology.orchestra.run.smithi076:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2023-12-10T18:51:16.893 INFO:teuthology.misc.health.smithi076.stdout: 2023-12-10T18:51:16.893 INFO:teuthology.misc.health.smithi076.stdout:{"epoch":7,"fsid":"af537260-6e4d-44cb-9e7e-e1ba6e0ece8e","created":"2023-12-10T18:51:08.563100+0000","modified":"2023-12-10T18:51:15.520930+0000","last_up_change":"0.000000","last_in_change":"2023-12-10T18:51:09.458539+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f61274b1-3e55-4802-a1da-3728a61b0bf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"7f08d0f1-6812-485c-9efa-0741603162e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8fd7962b-5418-4161-98a5-16a26bf0e050","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-10T18:51:16.918 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-12-10T18:51:17.496 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr:2023-12-10T18:51:17.490+0000 7ff7b3e0a640 -1 osd.0 0 waiting for initial osdmap 2023-12-10T18:51:17.517 INFO:tasks.ceph.cluster2.osd.2.smithi076.stderr:2023-12-10T18:51:17.514+0000 7f7f79375640 -1 osd.2 0 waiting for initial osdmap 2023-12-10T18:51:17.525 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr:2023-12-10T18:51:17.522+0000 7ff7ac457640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-10T18:51:17.554 INFO:tasks.ceph.cluster2.osd.2.smithi076.stderr:2023-12-10T18:51:17.550+0000 7f7f7018c640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-10T18:51:17.568 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr:2023-12-10T18:51:17.562+0000 7fa37f3b6640 -1 osd.1 0 waiting for initial osdmap 2023-12-10T18:51:17.605 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr:2023-12-10T18:51:17.602+0000 7fa377a03640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-10T18:51:23.221 DEBUG:teuthology.orchestra.run.smithi076:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2023-12-10T18:51:23.510 INFO:teuthology.misc.health.smithi076.stdout: 2023-12-10T18:51:23.510 INFO:teuthology.misc.health.smithi076.stdout:{"epoch":12,"fsid":"af537260-6e4d-44cb-9e7e-e1ba6e0ece8e","created":"2023-12-10T18:51:08.563100+0000","modified":"2023-12-10T18:51:22.593122+0000","last_up_change":"2023-12-10T18:51:18.518500+0000","last_in_change":"2023-12-10T18:51:09.458539+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-10T18:51:20.115438+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f61274b1-3e55-4802-a1da-3728a61b0bf5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":3457246957},{"type":"v1","addr":"172.21.15.76:6809","nonce":3457246957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":3457246957},{"type":"v1","addr":"172.21.15.76:6814","nonce":3457246957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6821","nonce":3457246957},{"type":"v1","addr":"172.21.15.76:6823","nonce":3457246957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":3457246957},{"type":"v1","addr":"172.21.15.76:6819","nonce":3457246957}]},"public_addr":"172.21.15.76:6809/3457246957","cluster_addr":"172.21.15.76:6814/3457246957","heartbeat_back_addr":"172.21.15.76:6823/3457246957","heartbeat_front_addr":"172.21.15.76:6819/3457246957","state":["exists","up"]},{"osd":1,"uuid":"7f08d0f1-6812-485c-9efa-0741603162e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6800","nonce":144319191},{"type":"v1","addr":"172.21.15.76:6801","nonce":144319191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":144319191},{"type":"v1","addr":"172.21.15.76:6803","nonce":144319191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6807","nonce":144319191},{"type":"v1","addr":"172.21.15.76:6810","nonce":144319191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":144319191},{"type":"v1","addr":"172.21.15.76:6805","nonce":144319191}]},"public_addr":"172.21.15.76:6801/144319191","cluster_addr":"172.21.15.76:6803/144319191","heartbeat_back_addr":"172.21.15.76:6810/144319191","heartbeat_front_addr":"172.21.15.76:6805/144319191","state":["exists","up"]},{"osd":2,"uuid":"8fd7962b-5418-4161-98a5-16a26bf0e050","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":1655581503},{"type":"v1","addr":"172.21.15.76:6811","nonce":1655581503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6813","nonce":1655581503},{"type":"v1","addr":"172.21.15.76:6815","nonce":1655581503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":1655581503},{"type":"v1","addr":"172.21.15.76:6822","nonce":1655581503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6817","nonce":1655581503},{"type":"v1","addr":"172.21.15.76:6818","nonce":1655581503}]},"public_addr":"172.21.15.76:6811/1655581503","cluster_addr":"172.21.15.76:6815/1655581503","heartbeat_back_addr":"172.21.15.76:6822/1655581503","heartbeat_front_addr":"172.21.15.76:6818/1655581503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-10T18:51:14.991448+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-10T18:51:14.818316+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-10T18:51:14.931679+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-10T18:51:23.527 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-12-10T18:51:23.527 INFO:tasks.ceph:Creating RBD pool 2023-12-10T18:51:23.527 DEBUG:teuthology.orchestra.run.smithi076:> sudo ceph --cluster cluster2 osd pool create rbd 8 2023-12-10T18:51:24.605 INFO:teuthology.orchestra.run.smithi076.stderr:pool 'rbd' created 2023-12-10T18:51:24.633 DEBUG:teuthology.orchestra.run.smithi076:> sudo ceph --cluster cluster2 osd pool application enable rbd rbd --yes-i-really-mean-it 2023-12-10T18:51:25.614 INFO:teuthology.orchestra.run.smithi076.stderr:enabled application 'rbd' on pool 'rbd' 2023-12-10T18:51:25.643 INFO:tasks.ceph:Starting mds daemons in cluster cluster2... 2023-12-10T18:51:25.643 INFO:tasks.ceph:Waiting until cluster2 daemons up and pgs clean... 2023-12-10T18:51:25.643 INFO:tasks.ceph.ceph_manager.cluster2:waiting for mgr available 2023-12-10T18:51:25.644 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 mgr dump --format=json 2023-12-10T18:51:25.644 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-12-10T18:51:25.976 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-10T18:51:25.992 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":5,"active_gid":4099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":1123043099},{"type":"v1","addr":"172.21.15.33:6827","nonce":1123043099}]},"active_addr":"172.21.15.33:6827/1123043099","active_change":"2023-12-10T18:51:12.027969+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.33:0","nonce":1172721146}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.33:0","nonce":1098522628}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.33:0","nonce":160888482}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.33:0","nonce":295598834}]}]} 2023-12-10T18:51:25.994 INFO:tasks.ceph.ceph_manager.cluster2:mgr available! 2023-12-10T18:51:25.995 INFO:tasks.ceph.ceph_manager.cluster2:waiting for all up 2023-12-10T18:51:25.995 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2023-12-10T18:51:26.281 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-10T18:51:26.282 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":14,"fsid":"af537260-6e4d-44cb-9e7e-e1ba6e0ece8e","created":"2023-12-10T18:51:08.563100+0000","modified":"2023-12-10T18:51:25.604213+0000","last_up_change":"2023-12-10T18:51:18.518500+0000","last_in_change":"2023-12-10T18:51:09.458539+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-10T18:51:20.115438+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-10T18:51:23.844501+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f61274b1-3e55-4802-a1da-3728a61b0bf5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":3457246957},{"type":"v1","addr":"172.21.15.76:6809","nonce":3457246957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":3457246957},{"type":"v1","addr":"172.21.15.76:6814","nonce":3457246957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6821","nonce":3457246957},{"type":"v1","addr":"172.21.15.76:6823","nonce":3457246957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":3457246957},{"type":"v1","addr":"172.21.15.76:6819","nonce":3457246957}]},"public_addr":"172.21.15.76:6809/3457246957","cluster_addr":"172.21.15.76:6814/3457246957","heartbeat_back_addr":"172.21.15.76:6823/3457246957","heartbeat_front_addr":"172.21.15.76:6819/3457246957","state":["exists","up"]},{"osd":1,"uuid":"7f08d0f1-6812-485c-9efa-0741603162e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6800","nonce":144319191},{"type":"v1","addr":"172.21.15.76:6801","nonce":144319191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":144319191},{"type":"v1","addr":"172.21.15.76:6803","nonce":144319191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6807","nonce":144319191},{"type":"v1","addr":"172.21.15.76:6810","nonce":144319191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":144319191},{"type":"v1","addr":"172.21.15.76:6805","nonce":144319191}]},"public_addr":"172.21.15.76:6801/144319191","cluster_addr":"172.21.15.76:6803/144319191","heartbeat_back_addr":"172.21.15.76:6810/144319191","heartbeat_front_addr":"172.21.15.76:6805/144319191","state":["exists","up"]},{"osd":2,"uuid":"8fd7962b-5418-4161-98a5-16a26bf0e050","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":1655581503},{"type":"v1","addr":"172.21.15.76:6811","nonce":1655581503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6813","nonce":1655581503},{"type":"v1","addr":"172.21.15.76:6815","nonce":1655581503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":1655581503},{"type":"v1","addr":"172.21.15.76:6822","nonce":1655581503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6817","nonce":1655581503},{"type":"v1","addr":"172.21.15.76:6818","nonce":1655581503}]},"public_addr":"172.21.15.76:6811/1655581503","cluster_addr":"172.21.15.76:6815/1655581503","heartbeat_back_addr":"172.21.15.76:6822/1655581503","heartbeat_front_addr":"172.21.15.76:6818/1655581503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-10T18:51:14.991448+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-10T18:51:14.818316+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-10T18:51:14.931679+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-10T18:51:26.298 INFO:tasks.ceph.ceph_manager.cluster2:all up! 2023-12-10T18:51:26.298 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2023-12-10T18:51:26.578 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-10T18:51:26.579 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":14,"fsid":"af537260-6e4d-44cb-9e7e-e1ba6e0ece8e","created":"2023-12-10T18:51:08.563100+0000","modified":"2023-12-10T18:51:25.604213+0000","last_up_change":"2023-12-10T18:51:18.518500+0000","last_in_change":"2023-12-10T18:51:09.458539+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-10T18:51:20.115438+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-10T18:51:23.844501+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f61274b1-3e55-4802-a1da-3728a61b0bf5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":3457246957},{"type":"v1","addr":"172.21.15.76:6809","nonce":3457246957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":3457246957},{"type":"v1","addr":"172.21.15.76:6814","nonce":3457246957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6821","nonce":3457246957},{"type":"v1","addr":"172.21.15.76:6823","nonce":3457246957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":3457246957},{"type":"v1","addr":"172.21.15.76:6819","nonce":3457246957}]},"public_addr":"172.21.15.76:6809/3457246957","cluster_addr":"172.21.15.76:6814/3457246957","heartbeat_back_addr":"172.21.15.76:6823/3457246957","heartbeat_front_addr":"172.21.15.76:6819/3457246957","state":["exists","up"]},{"osd":1,"uuid":"7f08d0f1-6812-485c-9efa-0741603162e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6800","nonce":144319191},{"type":"v1","addr":"172.21.15.76:6801","nonce":144319191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":144319191},{"type":"v1","addr":"172.21.15.76:6803","nonce":144319191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6807","nonce":144319191},{"type":"v1","addr":"172.21.15.76:6810","nonce":144319191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":144319191},{"type":"v1","addr":"172.21.15.76:6805","nonce":144319191}]},"public_addr":"172.21.15.76:6801/144319191","cluster_addr":"172.21.15.76:6803/144319191","heartbeat_back_addr":"172.21.15.76:6810/144319191","heartbeat_front_addr":"172.21.15.76:6805/144319191","state":["exists","up"]},{"osd":2,"uuid":"8fd7962b-5418-4161-98a5-16a26bf0e050","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":1655581503},{"type":"v1","addr":"172.21.15.76:6811","nonce":1655581503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6813","nonce":1655581503},{"type":"v1","addr":"172.21.15.76:6815","nonce":1655581503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":1655581503},{"type":"v1","addr":"172.21.15.76:6822","nonce":1655581503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6817","nonce":1655581503},{"type":"v1","addr":"172.21.15.76:6818","nonce":1655581503}]},"public_addr":"172.21.15.76:6811/1655581503","cluster_addr":"172.21.15.76:6815/1655581503","heartbeat_back_addr":"172.21.15.76:6822/1655581503","heartbeat_front_addr":"172.21.15.76:6818/1655581503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-10T18:51:14.991448+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-10T18:51:14.818316+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-10T18:51:14.931679+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-10T18:51:26.596 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 flush_pg_stats 2023-12-10T18:51:26.597 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 flush_pg_stats 2023-12-10T18:51:26.597 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 flush_pg_stats 2023-12-10T18:51:26.757 INFO:teuthology.orchestra.run.smithi076.stdout:34359738373 2023-12-10T18:51:26.757 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2023-12-10T18:51:26.758 INFO:teuthology.orchestra.run.smithi076.stdout:34359738374 2023-12-10T18:51:26.758 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2023-12-10T18:51:26.759 INFO:teuthology.orchestra.run.smithi076.stdout:34359738373 2023-12-10T18:51:26.760 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2023-12-10T18:51:27.080 INFO:teuthology.orchestra.run.smithi076.stdout:34359738372 2023-12-10T18:51:27.096 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738372 for osd.2 2023-12-10T18:51:27.099 INFO:teuthology.orchestra.run.smithi076.stdout:34359738372 2023-12-10T18:51:27.104 INFO:teuthology.orchestra.run.smithi076.stdout:34359738373 2023-12-10T18:51:27.118 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738372 for osd.0 2023-12-10T18:51:27.121 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.1 2023-12-10T18:51:28.098 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2023-12-10T18:51:28.118 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2023-12-10T18:51:28.122 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2023-12-10T18:51:28.415 INFO:teuthology.orchestra.run.smithi076.stdout:34359738372 2023-12-10T18:51:28.430 INFO:teuthology.orchestra.run.smithi076.stdout:34359738373 2023-12-10T18:51:28.431 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738372 for osd.2 2023-12-10T18:51:28.434 INFO:teuthology.orchestra.run.smithi076.stdout:34359738372 2023-12-10T18:51:28.449 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.1 2023-12-10T18:51:28.452 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738372 for osd.0 2023-12-10T18:51:29.432 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2023-12-10T18:51:29.450 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2023-12-10T18:51:29.454 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2023-12-10T18:51:29.729 INFO:teuthology.orchestra.run.smithi076.stdout:34359738373 2023-12-10T18:51:29.746 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738373 for osd.2 2023-12-10T18:51:29.747 DEBUG:teuthology.parallel:result is None 2023-12-10T18:51:29.756 INFO:teuthology.orchestra.run.smithi076.stdout:34359738373 2023-12-10T18:51:29.759 INFO:teuthology.orchestra.run.smithi076.stdout:34359738374 2023-12-10T18:51:29.780 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738373 for osd.0 2023-12-10T18:51:29.781 DEBUG:teuthology.parallel:result is None 2023-12-10T18:51:29.781 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738374 for osd.1 2023-12-10T18:51:29.781 DEBUG:teuthology.parallel:result is None 2023-12-10T18:51:29.782 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2023-12-10T18:51:29.782 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-10T18:51:30.099 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-10T18:51:30.099 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2023-12-10T18:51:30.115 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2023-12-10T18:51:30.042367+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82608,"kb_used_data":1544,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032912,"statfs":{"total":289910292480,"available":289825701888,"internally_reserved":0,"allocated":1581056,"data_stored":1292366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001576"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-10T18:51:26.624085+0000","last_change":"2023-12-10T18:51:25.628475+0000","last_active":"2023-12-10T18:51:26.624085+0000","last_peered":"2023-12-10T18:51:26.624085+0000","last_clean":"2023-12-10T18:51:26.624085+0000","last_became_active":"2023-12-10T18:51:25.628183+0000","last_became_peered":"2023-12-10T18:51:25.628183+0000","last_unstale":"2023-12-10T18:51:26.624085+0000","last_undegraded":"2023-12-10T18:51:26.624085+0000","last_fullsized":"2023-12-10T18:51:26.624085+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-10T18:51:24.596512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-10T18:51:24.596512+0000","last_clean_scrub_stamp":"2023-12-10T18:51:24.596512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-11T21:34:12.034203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-10T18:51:26.624019+0000","last_change":"2023-12-10T18:51:25.632802+0000","last_active":"2023-12-10T18:51:26.624019+0000","last_peered":"2023-12-10T18:51:26.624019+0000","last_clean":"2023-12-10T18:51:26.624019+0000","last_became_active":"2023-12-10T18:51:25.632411+0000","last_became_peered":"2023-12-10T18:51:25.632411+0000","last_unstale":"2023-12-10T18:51:26.624019+0000","last_undegraded":"2023-12-10T18:51:26.624019+0000","last_fullsized":"2023-12-10T18:51:26.624019+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-10T18:51:24.596512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-10T18:51:24.596512+0000","last_clean_scrub_stamp":"2023-12-10T18:51:24.596512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T03:50:38.823309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-10T18:51:26.624807+0000","last_change":"2023-12-10T18:51:25.633020+0000","last_active":"2023-12-10T18:51:26.624807+0000","last_peered":"2023-12-10T18:51:26.624807+0000","last_clean":"2023-12-10T18:51:26.624807+0000","last_became_active":"2023-12-10T18:51:25.632408+0000","last_became_peered":"2023-12-10T18:51:25.632408+0000","last_unstale":"2023-12-10T18:51:26.624807+0000","last_undegraded":"2023-12-10T18:51:26.624807+0000","last_fullsized":"2023-12-10T18:51:26.624807+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-10T18:51:24.596512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-10T18:51:24.596512+0000","last_clean_scrub_stamp":"2023-12-10T18:51:24.596512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T04:20:12.256921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-10T18:51:26.623957+0000","last_change":"2023-12-10T18:51:25.629010+0000","last_active":"2023-12-10T18:51:26.623957+0000","last_peered":"2023-12-10T18:51:26.623957+0000","last_clean":"2023-12-10T18:51:26.623957+0000","last_became_active":"2023-12-10T18:51:25.628717+0000","last_became_peered":"2023-12-10T18:51:25.628717+0000","last_unstale":"2023-12-10T18:51:26.623957+0000","last_undegraded":"2023-12-10T18:51:26.623957+0000","last_fullsized":"2023-12-10T18:51:26.623957+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-10T18:51:24.596512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-10T18:51:24.596512+0000","last_clean_scrub_stamp":"2023-12-10T18:51:24.596512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-11T23:34:57.110479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-10T18:51:27.278221+0000","last_change":"2023-12-10T18:51:25.631818+0000","last_active":"2023-12-10T18:51:27.278221+0000","last_peered":"2023-12-10T18:51:27.278221+0000","last_clean":"2023-12-10T18:51:27.278221+0000","last_became_active":"2023-12-10T18:51:25.631389+0000","last_became_peered":"2023-12-10T18:51:25.631389+0000","last_unstale":"2023-12-10T18:51:27.278221+0000","last_undegraded":"2023-12-10T18:51:27.278221+0000","last_fullsized":"2023-12-10T18:51:27.278221+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-10T18:51:24.596512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-10T18:51:24.596512+0000","last_clean_scrub_stamp":"2023-12-10T18:51:24.596512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-11T23:58:50.346741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-10T18:51:27.784198+0000","last_change":"2023-12-10T18:51:25.629603+0000","last_active":"2023-12-10T18:51:27.784198+0000","last_peered":"2023-12-10T18:51:27.784198+0000","last_clean":"2023-12-10T18:51:27.784198+0000","last_became_active":"2023-12-10T18:51:25.628950+0000","last_became_peered":"2023-12-10T18:51:25.628950+0000","last_unstale":"2023-12-10T18:51:27.784198+0000","last_undegraded":"2023-12-10T18:51:27.784198+0000","last_fullsized":"2023-12-10T18:51:27.784198+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-10T18:51:24.596512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-10T18:51:24.596512+0000","last_clean_scrub_stamp":"2023-12-10T18:51:24.596512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-11T22:31:25.360221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-10T18:51:27.784198+0000","last_change":"2023-12-10T18:51:25.631055+0000","last_active":"2023-12-10T18:51:27.784198+0000","last_peered":"2023-12-10T18:51:27.784198+0000","last_clean":"2023-12-10T18:51:27.784198+0000","last_became_active":"2023-12-10T18:51:25.630734+0000","last_became_peered":"2023-12-10T18:51:25.630734+0000","last_unstale":"2023-12-10T18:51:27.784198+0000","last_undegraded":"2023-12-10T18:51:27.784198+0000","last_fullsized":"2023-12-10T18:51:27.784198+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-10T18:51:24.596512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-10T18:51:24.596512+0000","last_clean_scrub_stamp":"2023-12-10T18:51:24.596512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-11T19:54:55.087633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-10T18:51:26.623950+0000","last_change":"2023-12-10T18:51:25.628613+0000","last_active":"2023-12-10T18:51:26.623950+0000","last_peered":"2023-12-10T18:51:26.623950+0000","last_clean":"2023-12-10T18:51:26.623950+0000","last_became_active":"2023-12-10T18:51:25.628218+0000","last_became_peered":"2023-12-10T18:51:25.628218+0000","last_unstale":"2023-12-10T18:51:26.623950+0000","last_undegraded":"2023-12-10T18:51:26.623950+0000","last_fullsized":"2023-12-10T18:51:26.623950+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-10T18:51:24.596512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-10T18:51:24.596512+0000","last_clean_scrub_stamp":"2023-12-10T18:51:24.596512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T02:43:41.886234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":141,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-10T18:51:25.624705+0000","last_change":"2023-12-10T18:51:21.599753+0000","last_active":"2023-12-10T18:51:25.624705+0000","last_peered":"2023-12-10T18:51:25.624705+0000","last_clean":"2023-12-10T18:51:25.624705+0000","last_became_active":"2023-12-10T18:51:21.598903+0000","last_became_peered":"2023-12-10T18:51:21.598903+0000","last_unstale":"2023-12-10T18:51:25.624705+0000","last_undegraded":"2023-12-10T18:51:25.624705+0000","last_fullsized":"2023-12-10T18:51:25.624705+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-10T18:51:20.578269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-10T18:51:20.578269+0000","last_clean_scrub_stamp":"2023-12-10T18:51:20.578269+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T00:34:13.476085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27152,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344688,"statfs":{"total":96636764160,"available":96608960512,"internally_reserved":0,"allocated":131072,"data_stored":37210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67200000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61599999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738375,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27728,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344112,"statfs":{"total":96636764160,"available":96608370688,"internally_reserved":0,"allocated":724992,"data_stored":627578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46600000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.309}]}]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27728,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344112,"statfs":{"total":96636764160,"available":96608370688,"internally_reserved":0,"allocated":724992,"data_stored":627578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.435}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-10T18:51:30.116 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-10T18:51:30.374 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-10T18:51:30.375 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2023-12-10T18:51:30.391 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2023-12-10T18:51:30.042367+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82608,"kb_used_data":1544,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032912,"statfs":{"total":289910292480,"available":289825701888,"internally_reserved":0,"allocated":1581056,"data_stored":1292366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001576"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-10T18:51:26.624085+0000","last_change":"2023-12-10T18:51:25.628475+0000","last_active":"2023-12-10T18:51:26.624085+0000","last_peered":"2023-12-10T18:51:26.624085+0000","last_clean":"2023-12-10T18:51:26.624085+0000","last_became_active":"2023-12-10T18:51:25.628183+0000","last_became_peered":"2023-12-10T18:51:25.628183+0000","last_unstale":"2023-12-10T18:51:26.624085+0000","last_undegraded":"2023-12-10T18:51:26.624085+0000","last_fullsized":"2023-12-10T18:51:26.624085+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-10T18:51:24.596512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-10T18:51:24.596512+0000","last_clean_scrub_stamp":"2023-12-10T18:51:24.596512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-11T21:34:12.034203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-10T18:51:26.624019+0000","last_change":"2023-12-10T18:51:25.632802+0000","last_active":"2023-12-10T18:51:26.624019+0000","last_peered":"2023-12-10T18:51:26.624019+0000","last_clean":"2023-12-10T18:51:26.624019+0000","last_became_active":"2023-12-10T18:51:25.632411+0000","last_became_peered":"2023-12-10T18:51:25.632411+0000","last_unstale":"2023-12-10T18:51:26.624019+0000","last_undegraded":"2023-12-10T18:51:26.624019+0000","last_fullsized":"2023-12-10T18:51:26.624019+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-10T18:51:24.596512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-10T18:51:24.596512+0000","last_clean_scrub_stamp":"2023-12-10T18:51:24.596512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T03:50:38.823309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-10T18:51:26.624807+0000","last_change":"2023-12-10T18:51:25.633020+0000","last_active":"2023-12-10T18:51:26.624807+0000","last_peered":"2023-12-10T18:51:26.624807+0000","last_clean":"2023-12-10T18:51:26.624807+0000","last_became_active":"2023-12-10T18:51:25.632408+0000","last_became_peered":"2023-12-10T18:51:25.632408+0000","last_unstale":"2023-12-10T18:51:26.624807+0000","last_undegraded":"2023-12-10T18:51:26.624807+0000","last_fullsized":"2023-12-10T18:51:26.624807+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-10T18:51:24.596512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-10T18:51:24.596512+0000","last_clean_scrub_stamp":"2023-12-10T18:51:24.596512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T04:20:12.256921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-10T18:51:26.623957+0000","last_change":"2023-12-10T18:51:25.629010+0000","last_active":"2023-12-10T18:51:26.623957+0000","last_peered":"2023-12-10T18:51:26.623957+0000","last_clean":"2023-12-10T18:51:26.623957+0000","last_became_active":"2023-12-10T18:51:25.628717+0000","last_became_peered":"2023-12-10T18:51:25.628717+0000","last_unstale":"2023-12-10T18:51:26.623957+0000","last_undegraded":"2023-12-10T18:51:26.623957+0000","last_fullsized":"2023-12-10T18:51:26.623957+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-10T18:51:24.596512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-10T18:51:24.596512+0000","last_clean_scrub_stamp":"2023-12-10T18:51:24.596512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-11T23:34:57.110479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-10T18:51:27.278221+0000","last_change":"2023-12-10T18:51:25.631818+0000","last_active":"2023-12-10T18:51:27.278221+0000","last_peered":"2023-12-10T18:51:27.278221+0000","last_clean":"2023-12-10T18:51:27.278221+0000","last_became_active":"2023-12-10T18:51:25.631389+0000","last_became_peered":"2023-12-10T18:51:25.631389+0000","last_unstale":"2023-12-10T18:51:27.278221+0000","last_undegraded":"2023-12-10T18:51:27.278221+0000","last_fullsized":"2023-12-10T18:51:27.278221+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-10T18:51:24.596512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-10T18:51:24.596512+0000","last_clean_scrub_stamp":"2023-12-10T18:51:24.596512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-11T23:58:50.346741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-10T18:51:27.784198+0000","last_change":"2023-12-10T18:51:25.629603+0000","last_active":"2023-12-10T18:51:27.784198+0000","last_peered":"2023-12-10T18:51:27.784198+0000","last_clean":"2023-12-10T18:51:27.784198+0000","last_became_active":"2023-12-10T18:51:25.628950+0000","last_became_peered":"2023-12-10T18:51:25.628950+0000","last_unstale":"2023-12-10T18:51:27.784198+0000","last_undegraded":"2023-12-10T18:51:27.784198+0000","last_fullsized":"2023-12-10T18:51:27.784198+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-10T18:51:24.596512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-10T18:51:24.596512+0000","last_clean_scrub_stamp":"2023-12-10T18:51:24.596512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-11T22:31:25.360221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-10T18:51:27.784198+0000","last_change":"2023-12-10T18:51:25.631055+0000","last_active":"2023-12-10T18:51:27.784198+0000","last_peered":"2023-12-10T18:51:27.784198+0000","last_clean":"2023-12-10T18:51:27.784198+0000","last_became_active":"2023-12-10T18:51:25.630734+0000","last_became_peered":"2023-12-10T18:51:25.630734+0000","last_unstale":"2023-12-10T18:51:27.784198+0000","last_undegraded":"2023-12-10T18:51:27.784198+0000","last_fullsized":"2023-12-10T18:51:27.784198+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-10T18:51:24.596512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-10T18:51:24.596512+0000","last_clean_scrub_stamp":"2023-12-10T18:51:24.596512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-11T19:54:55.087633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-10T18:51:26.623950+0000","last_change":"2023-12-10T18:51:25.628613+0000","last_active":"2023-12-10T18:51:26.623950+0000","last_peered":"2023-12-10T18:51:26.623950+0000","last_clean":"2023-12-10T18:51:26.623950+0000","last_became_active":"2023-12-10T18:51:25.628218+0000","last_became_peered":"2023-12-10T18:51:25.628218+0000","last_unstale":"2023-12-10T18:51:26.623950+0000","last_undegraded":"2023-12-10T18:51:26.623950+0000","last_fullsized":"2023-12-10T18:51:26.623950+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-10T18:51:24.596512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-10T18:51:24.596512+0000","last_clean_scrub_stamp":"2023-12-10T18:51:24.596512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T02:43:41.886234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":141,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-10T18:51:25.624705+0000","last_change":"2023-12-10T18:51:21.599753+0000","last_active":"2023-12-10T18:51:25.624705+0000","last_peered":"2023-12-10T18:51:25.624705+0000","last_clean":"2023-12-10T18:51:25.624705+0000","last_became_active":"2023-12-10T18:51:21.598903+0000","last_became_peered":"2023-12-10T18:51:21.598903+0000","last_unstale":"2023-12-10T18:51:25.624705+0000","last_undegraded":"2023-12-10T18:51:25.624705+0000","last_fullsized":"2023-12-10T18:51:25.624705+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-10T18:51:20.578269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-10T18:51:20.578269+0000","last_clean_scrub_stamp":"2023-12-10T18:51:20.578269+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-12T00:34:13.476085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27152,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344688,"statfs":{"total":96636764160,"available":96608960512,"internally_reserved":0,"allocated":131072,"data_stored":37210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67200000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61599999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738375,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27728,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344112,"statfs":{"total":96636764160,"available":96608370688,"internally_reserved":0,"allocated":724992,"data_stored":627578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46600000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.309}]}]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27728,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344112,"statfs":{"total":96636764160,"available":96608370688,"internally_reserved":0,"allocated":724992,"data_stored":627578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.435}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-10T18:51:30.392 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2023-12-10T18:51:30.392 INFO:tasks.ceph:Waiting until ceph cluster cluster2 is healthy... 2023-12-10T18:51:30.392 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy 2023-12-10T18:51:30.392 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 health --format=json 2023-12-10T18:51:30.708 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-10T18:51:30.708 INFO:teuthology.orchestra.run.smithi076.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-12-10T18:51:30.726 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy done 2023-12-10T18:51:30.727 INFO:teuthology.run_tasks:Running task exec... 2023-12-10T18:51:30.736 INFO:teuthology.task.exec:Executing custom commands... 2023-12-10T18:51:30.737 INFO:teuthology.task.exec:Running commands on role cluster1.client.mirror.0 host ubuntu@smithi076.front.sepia.ceph.com 2023-12-10T18:51:30.737 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2023-12-10T18:51:31.113 INFO:teuthology.orchestra.run.smithi076.stdout:[client.mirror] 2023-12-10T18:51:31.113 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQBpCHZlukn5IhAAVam+mhPU05gN36oBgaBbag== 2023-12-10T18:51:31.113 INFO:teuthology.orchestra.run.smithi076.stdout: caps mon = "profile rbd-mirror-peer" 2023-12-10T18:51:31.113 INFO:teuthology.orchestra.run.smithi076.stdout: caps osd = "profile rbd" 2023-12-10T18:51:31.113 INFO:teuthology.orchestra.run.smithi076.stderr:updated caps for client.mirror 2023-12-10T18:51:31.132 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-10T18:51:31.491 INFO:teuthology.orchestra.run.smithi076.stdout:[client.mirror.0] 2023-12-10T18:51:31.491 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQBpCHZlyyqVKhAAif2T0ChYfzY/KMnp9Q35Sg== 2023-12-10T18:51:31.491 INFO:teuthology.orchestra.run.smithi076.stdout: caps mon = "profile rbd-mirror" 2023-12-10T18:51:31.491 INFO:teuthology.orchestra.run.smithi076.stdout: caps osd = "profile rbd" 2023-12-10T18:51:31.491 INFO:teuthology.orchestra.run.smithi076.stderr:updated caps for client.mirror.0 2023-12-10T18:51:31.509 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-10T18:51:31.852 INFO:teuthology.orchestra.run.smithi076.stdout:[client.mirror.1] 2023-12-10T18:51:31.852 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQBpCHZlzXz6LxAAVd7Uh4mATM05DtSjlYthJw== 2023-12-10T18:51:31.852 INFO:teuthology.orchestra.run.smithi076.stdout: caps mon = "profile rbd-mirror" 2023-12-10T18:51:31.852 INFO:teuthology.orchestra.run.smithi076.stdout: caps osd = "profile rbd" 2023-12-10T18:51:31.852 INFO:teuthology.orchestra.run.smithi076.stderr:updated caps for client.mirror.1 2023-12-10T18:51:31.870 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-10T18:51:32.225 INFO:teuthology.orchestra.run.smithi076.stdout:[client.mirror.2] 2023-12-10T18:51:32.225 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQBpCHZlNuQbNRAA/dGJQv/ryls/ICFExzk+Gg== 2023-12-10T18:51:32.225 INFO:teuthology.orchestra.run.smithi076.stdout: caps mon = "profile rbd-mirror" 2023-12-10T18:51:32.225 INFO:teuthology.orchestra.run.smithi076.stdout: caps osd = "profile rbd" 2023-12-10T18:51:32.225 INFO:teuthology.orchestra.run.smithi076.stderr:updated caps for client.mirror.2 2023-12-10T18:51:32.243 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-10T18:51:32.598 INFO:teuthology.orchestra.run.smithi076.stdout:[client.mirror.3] 2023-12-10T18:51:32.598 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQBpCHZlt+IGOhAAJ5QPB/ViM7rmYVPcEM374w== 2023-12-10T18:51:32.598 INFO:teuthology.orchestra.run.smithi076.stdout: caps mon = "profile rbd-mirror" 2023-12-10T18:51:32.598 INFO:teuthology.orchestra.run.smithi076.stdout: caps osd = "profile rbd" 2023-12-10T18:51:32.599 INFO:teuthology.orchestra.run.smithi076.stderr:updated caps for client.mirror.3 2023-12-10T18:51:32.616 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2023-12-10T18:51:32.940 INFO:teuthology.orchestra.run.smithi076.stdout:[client.mirror] 2023-12-10T18:51:32.940 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQCOCHZl5lM6GBAAy2ylKP0J9DtNfzabMmPHuA== 2023-12-10T18:51:32.940 INFO:teuthology.orchestra.run.smithi076.stdout: caps mon = "profile rbd-mirror-peer" 2023-12-10T18:51:32.940 INFO:teuthology.orchestra.run.smithi076.stdout: caps osd = "profile rbd" 2023-12-10T18:51:32.941 INFO:teuthology.orchestra.run.smithi076.stderr:updated caps for client.mirror 2023-12-10T18:51:32.957 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-10T18:51:33.318 INFO:teuthology.orchestra.run.smithi076.stdout:[client.mirror.0] 2023-12-10T18:51:33.318 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQCOCHZl8oU+HRAA+QwAThxlqSY6m3L1JmJP1Q== 2023-12-10T18:51:33.319 INFO:teuthology.orchestra.run.smithi076.stdout: caps mon = "profile rbd-mirror" 2023-12-10T18:51:33.319 INFO:teuthology.orchestra.run.smithi076.stdout: caps osd = "profile rbd" 2023-12-10T18:51:33.319 INFO:teuthology.orchestra.run.smithi076.stderr:updated caps for client.mirror.0 2023-12-10T18:51:33.337 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-10T18:51:33.697 INFO:teuthology.orchestra.run.smithi076.stdout:[client.mirror.1] 2023-12-10T18:51:33.697 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQCOCHZljsMXIhAA8P4oBaTgS6HWZ7xK00SEGA== 2023-12-10T18:51:33.697 INFO:teuthology.orchestra.run.smithi076.stdout: caps mon = "profile rbd-mirror" 2023-12-10T18:51:33.698 INFO:teuthology.orchestra.run.smithi076.stdout: caps osd = "profile rbd" 2023-12-10T18:51:33.698 INFO:teuthology.orchestra.run.smithi076.stderr:updated caps for client.mirror.1 2023-12-10T18:51:33.717 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-10T18:51:34.062 INFO:teuthology.orchestra.run.smithi076.stdout:[client.mirror.2] 2023-12-10T18:51:34.062 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQCOCHZlKZEXJxAA6PcTShEf7TZr/qupREEM0w== 2023-12-10T18:51:34.062 INFO:teuthology.orchestra.run.smithi076.stdout: caps mon = "profile rbd-mirror" 2023-12-10T18:51:34.063 INFO:teuthology.orchestra.run.smithi076.stdout: caps osd = "profile rbd" 2023-12-10T18:51:34.063 INFO:teuthology.orchestra.run.smithi076.stderr:updated caps for client.mirror.2 2023-12-10T18:51:34.080 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-10T18:51:34.429 INFO:teuthology.orchestra.run.smithi076.stdout:[client.mirror.3] 2023-12-10T18:51:34.429 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQCOCHZlIAQvLBAAZaPD6DfyCBBgDCL86s4irA== 2023-12-10T18:51:34.429 INFO:teuthology.orchestra.run.smithi076.stdout: caps mon = "profile rbd-mirror" 2023-12-10T18:51:34.429 INFO:teuthology.orchestra.run.smithi076.stdout: caps osd = "profile rbd" 2023-12-10T18:51:34.429 INFO:teuthology.orchestra.run.smithi076.stderr:updated caps for client.mirror.3 2023-12-10T18:51:34.449 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-10T18:51:34.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-10T18:51:34.461 DEBUG:teuthology.orchestra.run.smithi076:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-10T18:51:34.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-10T18:51:34.463 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-10T18:51:34.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-12-10T18:51:34.474 DEBUG:teuthology.orchestra.run.smithi076:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-12-10T18:51:34.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-12-10T18:51:34.477 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-10T18:51:34.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2023-12-10T18:51:34.485 DEBUG:teuthology.orchestra.run.smithi076:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2023-12-10T18:51:34.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2023-12-10T18:51:34.488 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-10T18:51:34.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2023-12-10T18:51:34.498 DEBUG:teuthology.orchestra.run.smithi076:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2023-12-10T18:51:34.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2023-12-10T18:51:34.500 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-10T18:51:34.509 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-12-10T18:51:34.509 DEBUG:teuthology.orchestra.run.smithi076:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-12-10T18:51:34.511 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-12-10T18:51:34.511 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-10T18:51:34.521 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-12-10T18:51:34.522 DEBUG:teuthology.orchestra.run.smithi076:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-12-10T18:51:34.525 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-12-10T18:51:34.525 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-10T18:51:34.536 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2023-12-10T18:51:34.537 DEBUG:teuthology.orchestra.run.smithi076:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2023-12-10T18:51:34.539 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2023-12-10T18:51:34.539 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-10T18:51:34.548 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2023-12-10T18:51:34.549 DEBUG:teuthology.orchestra.run.smithi076:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2023-12-10T18:51:34.551 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2023-12-10T18:51:34.551 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2023-12-10T18:51:34.562 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1702234294 2023-12-10T18:51:34.562 DEBUG:tasks.rbd_mirror_thrash:Yielding 2023-12-10T18:51:34.562 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2023-12-10T18:51:34.573 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1702234294 2023-12-10T18:51:34.573 DEBUG:tasks.rbd_mirror_thrash:Yielding 2023-12-10T18:51:34.573 INFO:teuthology.run_tasks:Running task workunit... 2023-12-10T18:51:34.586 INFO:tasks.workunit:Pulling workunits from ref 3d2db44fa991bc355074aa9b7647f63b77201fef 2023-12-10T18:51:34.587 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-12-10T18:51:34.587 DEBUG:teuthology.orchestra.run.smithi076:> stat -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2023-12-10T18:51:34.587 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster1 2023-12-10T18:51:34.588 INFO:tasks.rbd_mirror_thrash:waiting for 65 secs before thrashing 2023-12-10T18:51:34.588 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster2 2023-12-10T18:51:34.588 INFO:tasks.rbd_mirror_thrash:waiting for 72 secs before thrashing 2023-12-10T18:51:34.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-10T18:51:34.593 INFO:teuthology.orchestra.run.smithi076.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.cluster1.mirror': No such file or directory 2023-12-10T18:51:34.593 DEBUG:teuthology.orchestra.run.smithi076:> mkdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2023-12-10T18:51:34.644 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.cluster1.mirror 2023-12-10T18:51:34.644 DEBUG:teuthology.orchestra.run.smithi076:> cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror && mkdir -- client.mirror 2023-12-10T18:51:34.692 INFO:tasks.workunit:timeout=6h 2023-12-10T18:51:34.692 INFO:tasks.workunit:cleanup=True 2023-12-10T18:51:34.692 DEBUG:teuthology.orchestra.run.smithi076:> rm -rf /home/ubuntu/cephtest/clone.cluster1.client.mirror && git clone https://github.com/ceph/ceph-ci /home/ubuntu/cephtest/clone.cluster1.client.mirror && cd /home/ubuntu/cephtest/clone.cluster1.client.mirror && git checkout 3d2db44fa991bc355074aa9b7647f63b77201fef 2023-12-10T18:51:34.741 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Cloning into '/home/ubuntu/cephtest/clone.cluster1.client.mirror'... 2023-12-10T18:52:16.915 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Note: switching to '3d2db44fa991bc355074aa9b7647f63b77201fef'. 2023-12-10T18:52:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T18:52:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-12-10T18:52:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:changes and commit them, and you can discard any commits you make in this 2023-12-10T18:52:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:state without impacting any branches by switching back to a branch. 2023-12-10T18:52:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T18:52:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:If you want to create a new branch to retain commits you create, you may 2023-12-10T18:52:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:do so (now or later) by using -c with the switch command. Example: 2023-12-10T18:52:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T18:52:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: git switch -c 2023-12-10T18:52:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T18:52:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Or undo this operation with: 2023-12-10T18:52:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T18:52:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: git switch - 2023-12-10T18:52:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T18:52:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-12-10T18:52:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T18:52:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:HEAD is now at 3d2db44fa99 Merge branch 'rbd-external-migrate' of https://github.com/orozery/ceph into HEAD 2023-12-10T18:52:16.921 DEBUG:teuthology.orchestra.run.smithi076:> cd -- /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror 2023-12-10T18:52:16.970 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-12-10T18:52:16.974 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2023-12-10T18:52:16.974 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-12-10T18:52:17.796 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-12-10T18:52:17.872 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-12-10T18:52:17.920 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2023-12-10T18:52:17.922 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2023-12-10T18:52:17.922 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-12-10T18:52:17.971 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2023-12-10T18:52:17.974 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2023-12-10T18:52:17.974 DEBUG:teuthology.orchestra.run.smithi076:> dd if=/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror of=/dev/stdout 2023-12-10T18:52:18.022 INFO:tasks.workunit:Running workunits matching rbd/rbd_mirror_stress.sh on cluster1.client.mirror... 2023-12-10T18:52:18.023 INFO:tasks.workunit:Running workunit rbd/rbd_mirror_stress.sh... 2023-12-10T18:52:18.023 DEBUG:teuthology.orchestra.run.smithi076:workunit test rbd/rbd_mirror_stress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3d2db44fa991bc355074aa9b7647f63b77201fef TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS='' MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering RBD_MIRROR_INSTANCES=4 RBD_MIRROR_MODE=snapshot RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2023-12-10T18:52:18.074 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ IMAGE_COUNT=50 2023-12-10T18:52:18.074 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ export LOCKDEP=0 2023-12-10T18:52:18.074 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2023-12-10T18:52:18.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ . /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_helpers.sh 2023-12-10T18:52:18.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ type xmlstarlet 2023-12-10T18:52:18.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ XMLSTARLET=xmlstarlet 2023-12-10T18:52:18.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ RBD_MIRROR_INSTANCES=4 2023-12-10T18:52:18.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ CLUSTER1=cluster1 2023-12-10T18:52:18.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ CLUSTER2=cluster2 2023-12-10T18:52:18.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ PEER_CLUSTER_SUFFIX= 2023-12-10T18:52:18.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ POOL=mirror 2023-12-10T18:52:18.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ PARENT_POOL=mirror_parent 2023-12-10T18:52:18.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ NS1=ns1 2023-12-10T18:52:18.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ NS2=ns2 2023-12-10T18:52:18.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ TEMPDIR= 2023-12-10T18:52:18.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ CEPH_ID=mirror 2023-12-10T18:52:18.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ RBD_IMAGE_FEATURES=layering 2023-12-10T18:52:18.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ MIRROR_USER_ID_PREFIX=mirror. 2023-12-10T18:52:18.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ RBD_MIRROR_MODE=snapshot 2023-12-10T18:52:18.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ MIRROR_POOL_MODE=image 2023-12-10T18:52:18.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = snapshot ] 2023-12-10T18:52:18.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ MIRROR_POOL_MODE=image 2023-12-10T18:52:18.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ export CEPH_ARGS=--id mirror 2023-12-10T18:52:18.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ LAST_MIRROR_INSTANCE=3 2023-12-10T18:52:18.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2023-12-10T18:52:18.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ readlink -f /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/../../../src 2023-12-10T18:52:18.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror/src 2023-12-10T18:52:18.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ CEPH_BIN=. 2023-12-10T18:52:18.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ CEPH_SRC=. 2023-12-10T18:52:18.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -e CMakeCache.txt ] 2023-12-10T18:52:18.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 -gt 0 ] 2023-12-10T18:52:18.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ setup 2023-12-10T18:52:18.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local c 2023-12-10T18:52:18.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ trap cleanup $? INT TERM EXIT 2023-12-10T18:52:18.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ setup_tempdir 2023-12-10T18:52:18.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T18:52:18.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mktemp -d 2023-12-10T18:52:18.080 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ TEMPDIR=/tmp/tmp.qNjRyMq75D 2023-12-10T18:52:18.080 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -z 1 ] 2023-12-10T18:52:18.080 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ setup_pools cluster1 cluster2 2023-12-10T18:52:18.080 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:52:18.080 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local remote_cluster=cluster2 2023-12-10T18:52:18.080 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mon_map_file 2023-12-10T18:52:18.080 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mon_addr 2023-12-10T18:52:18.080 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local admin_key_file 2023-12-10T18:52:18.080 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local uuid 2023-12-10T18:52:18.080 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ CEPH_ARGS= ceph --cluster cluster1 osd pool create mirror 64 64 2023-12-10T18:52:19.033 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:pool 'mirror' already exists 2023-12-10T18:52:19.050 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ CEPH_ARGS= ceph --cluster cluster1 osd pool create mirror_parent 64 64 2023-12-10T18:52:20.044 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:pool 'mirror_parent' already exists 2023-12-10T18:52:20.062 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ CEPH_ARGS= rbd --cluster cluster1 pool init mirror 2023-12-10T18:52:23.045 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ CEPH_ARGS= rbd --cluster cluster1 pool init mirror_parent 2023-12-10T18:52:26.045 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T18:52:26.045 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror pool enable --site-name cluster1 mirror image 2023-12-10T18:52:26.111 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 mirror pool enable mirror_parent image 2023-12-10T18:52:26.169 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 namespace create mirror/ns1 2023-12-10T18:52:26.215 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 namespace create mirror/ns2 2023-12-10T18:52:26.261 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns1 image 2023-12-10T18:52:26.311 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns2 image 2023-12-10T18:52:26.366 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -z ] 2023-12-10T18:52:26.366 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -z ] 2023-12-10T18:52:26.366 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ peer_add cluster1 mirror cluster2 2023-12-10T18:52:26.366 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:52:26.366 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T18:52:26.366 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:52:26.366 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T18:52:26.366 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local client_cluster=cluster2 2023-12-10T18:52:26.366 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T18:52:26.366 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local remote_cluster=cluster2 2023-12-10T18:52:26.366 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local uuid_var_name 2023-12-10T18:52:26.366 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T18:52:26.367 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local error_code 2023-12-10T18:52:26.367 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local peer_uuid 2023-12-10T18:52:26.367 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T18:52:26.367 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 mirror pool peer add mirror cluster2 2023-12-10T18:52:26.411 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ peer_uuid=cea0e0df-3ba5-45fb-ab55-c5b809a43d2b 2023-12-10T18:52:26.411 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=0 2023-12-10T18:52:26.411 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T18:52:26.411 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 -eq 17 ] 2023-12-10T18:52:26.411 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 0 -eq 0 2023-12-10T18:52:26.411 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T18:52:26.411 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:52:26.411 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ peer_add cluster1 mirror_parent cluster2 2023-12-10T18:52:26.411 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:52:26.411 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T18:52:26.411 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror_parent 2023-12-10T18:52:26.412 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T18:52:26.412 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local client_cluster=cluster2 2023-12-10T18:52:26.412 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T18:52:26.412 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local remote_cluster=cluster2 2023-12-10T18:52:26.412 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local uuid_var_name 2023-12-10T18:52:26.412 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T18:52:26.412 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local error_code 2023-12-10T18:52:26.412 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local peer_uuid 2023-12-10T18:52:26.412 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T18:52:26.412 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 mirror pool peer add mirror_parent cluster2 2023-12-10T18:52:26.457 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ peer_uuid=0f54ea86-1117-4a80-9b51-0eebdfe1047a 2023-12-10T18:52:26.457 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=0 2023-12-10T18:52:26.457 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T18:52:26.458 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 -eq 17 ] 2023-12-10T18:52:26.458 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 0 -eq 0 2023-12-10T18:52:26.458 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T18:52:26.458 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:52:26.458 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ setup_pools cluster2 cluster1 2023-12-10T18:52:26.458 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T18:52:26.458 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local remote_cluster=cluster1 2023-12-10T18:52:26.458 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mon_map_file 2023-12-10T18:52:26.458 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mon_addr 2023-12-10T18:52:26.458 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local admin_key_file 2023-12-10T18:52:26.458 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local uuid 2023-12-10T18:52:26.459 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ CEPH_ARGS= ceph --cluster cluster2 osd pool create mirror 64 64 2023-12-10T18:52:27.701 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:pool 'mirror' already exists 2023-12-10T18:52:27.718 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ CEPH_ARGS= ceph --cluster cluster2 osd pool create mirror_parent 64 64 2023-12-10T18:52:28.735 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:pool 'mirror_parent' already exists 2023-12-10T18:52:28.750 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ CEPH_ARGS= rbd --cluster cluster2 pool init mirror 2023-12-10T18:52:31.665 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ CEPH_ARGS= rbd --cluster cluster2 pool init mirror_parent 2023-12-10T18:52:34.682 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T18:52:34.682 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ CEPH_ARGS= rbd --cluster cluster2 mirror pool enable --site-name cluster2 mirror image 2023-12-10T18:52:34.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr:2023-12-10T18:52:34.675+0000 7f73aaa2c880 -1 rbd::mirror::PoolReplayer: 0x562139a00c00 init: failed to initialize remote pool poller: (2) No such file or directory 2023-12-10T18:52:34.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi076.stderr:2023-12-10T18:52:34.675+0000 7fbfbd579880 -1 rbd::mirror::PoolReplayer: 0x55b940b2ac00 init: failed to initialize remote pool poller: (2) No such file or directory 2023-12-10T18:52:34.685 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi076.stderr:2023-12-10T18:52:34.675+0000 7f57bc51b880 -1 rbd::mirror::PoolReplayer: 0x56138fdeac00 init: failed to initialize remote pool poller: (2) No such file or directory 2023-12-10T18:52:34.691 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi076.stderr:2023-12-10T18:52:34.675+0000 7f432b9c1880 -1 rbd::mirror::PoolReplayer: 0x55af1f25ac00 init: failed to initialize remote pool poller: (2) No such file or directory 2023-12-10T18:52:34.734 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 mirror pool enable mirror_parent image 2023-12-10T18:52:34.760 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi076.stderr:2023-12-10T18:52:34.751+0000 7fbfbd579880 -1 rbd::mirror::PoolReplayer: 0x55b94183d500 init: failed to initialize remote pool poller: (2) No such file or directory 2023-12-10T18:52:34.779 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 namespace create mirror/ns1 2023-12-10T18:52:34.815 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 namespace create mirror/ns2 2023-12-10T18:52:34.852 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns1 image 2023-12-10T18:52:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns2 image 2023-12-10T18:52:34.934 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -z ] 2023-12-10T18:52:34.935 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -z ] 2023-12-10T18:52:34.935 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ peer_add cluster2 mirror cluster1 2023-12-10T18:52:34.935 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T18:52:34.935 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T18:52:34.935 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:52:34.935 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T18:52:34.935 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local client_cluster=cluster1 2023-12-10T18:52:34.935 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T18:52:34.935 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local remote_cluster=cluster1 2023-12-10T18:52:34.935 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local uuid_var_name 2023-12-10T18:52:34.935 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T18:52:34.935 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local error_code 2023-12-10T18:52:34.935 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local peer_uuid 2023-12-10T18:52:34.935 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T18:52:34.935 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 mirror pool peer add mirror cluster1 2023-12-10T18:52:34.976 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ peer_uuid=e89915c5-bc65-4c1f-8fd5-3de6a053be01 2023-12-10T18:52:34.976 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=0 2023-12-10T18:52:34.976 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T18:52:34.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 -eq 17 ] 2023-12-10T18:52:34.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 0 -eq 0 2023-12-10T18:52:34.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T18:52:34.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:52:34.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ peer_add cluster2 mirror_parent cluster1 2023-12-10T18:52:34.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T18:52:34.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T18:52:34.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror_parent 2023-12-10T18:52:34.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T18:52:34.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local client_cluster=cluster1 2023-12-10T18:52:34.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T18:52:34.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local remote_cluster=cluster1 2023-12-10T18:52:34.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local uuid_var_name 2023-12-10T18:52:34.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T18:52:34.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local error_code 2023-12-10T18:52:34.978 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local peer_uuid 2023-12-10T18:52:34.978 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T18:52:34.978 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 mirror pool peer add mirror_parent cluster1 2023-12-10T18:52:35.012 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: mirror peer already exists 2023-12-10T18:52:35.015 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ peer_uuid= 2023-12-10T18:52:35.015 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=17 2023-12-10T18:52:35.015 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T18:52:35.015 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 17 -eq 17 ] 2023-12-10T18:52:35.016 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T18:52:36.017 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd mirror pool info --cluster cluster2 --pool mirror_parent --format xml 2023-12-10T18:52:36.017 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //peers/peer[site_name='cluster1']/uuid 2023-12-10T18:52:36.080 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ peer_uuid=cc656db1-645c-422c-9461-7931a6f7f8b4 2023-12-10T18:52:36.081 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ CEPH_ARGS= rbd --cluster cluster2 --pool mirror_parent mirror pool peer remove cc656db1-645c-422c-9461-7931a6f7f8b4 2023-12-10T18:52:36.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T18:52:36.125 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 mirror pool peer add mirror_parent cluster1 2023-12-10T18:52:36.163 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ peer_uuid=e28b2ed8-4538-4522-867e-b36958a5a8a2 2023-12-10T18:52:36.163 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=0 2023-12-10T18:52:36.163 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T18:52:36.163 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 -eq 17 ] 2023-12-10T18:52:36.163 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 0 -eq 0 2023-12-10T18:52:36.163 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T18:52:36.163 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:52:36.163 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T18:52:36.163 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ start_mirrors cluster1 2023-12-10T18:52:36.163 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:52:36.163 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T18:52:36.174 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ start_mirror cluster1:0 2023-12-10T18:52:36.174 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T18:52:36.174 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:52:36.174 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T18:52:36.174 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T18:52:36.174 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T18:52:36.174 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:52:36.175 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:52:36.175 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T18:52:36.175 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T18:52:36.175 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T18:52:36.175 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:52:36.175 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T18:52:36.175 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T18:52:36.175 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 1 2023-12-10T18:52:36.175 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 2023-12-10T18:52:36.175 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ start_mirror cluster1:1 2023-12-10T18:52:36.175 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:1 2023-12-10T18:52:36.175 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:52:36.175 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-10T18:52:36.175 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T18:52:36.175 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T18:52:36.175 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:52:36.176 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:52:36.176 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T18:52:36.176 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T18:52:36.176 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T18:52:36.176 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:52:36.176 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T18:52:36.176 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T18:52:36.176 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 1 2023-12-10T18:52:36.176 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 2023-12-10T18:52:36.176 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ start_mirror cluster1:2 2023-12-10T18:52:36.176 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:2 2023-12-10T18:52:36.176 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:52:36.176 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-10T18:52:36.176 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T18:52:36.176 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T18:52:36.176 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:52:36.176 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:52:36.177 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T18:52:36.177 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T18:52:36.177 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T18:52:36.177 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:52:36.177 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T18:52:36.177 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T18:52:36.177 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 1 2023-12-10T18:52:36.177 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 2023-12-10T18:52:36.177 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ start_mirror cluster1:3 2023-12-10T18:52:36.177 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:3 2023-12-10T18:52:36.177 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:52:36.177 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-10T18:52:36.177 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:3 2023-12-10T18:52:36.177 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T18:52:36.177 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:52:36.177 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:52:36.178 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T18:52:36.178 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 3 = cluster1:3 ] 2023-12-10T18:52:36.178 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T18:52:36.178 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:52:36.178 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=3 2023-12-10T18:52:36.178 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T18:52:36.178 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 1 2023-12-10T18:52:36.178 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 2023-12-10T18:52:36.178 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ start_mirrors cluster2 2023-12-10T18:52:36.178 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T18:52:36.178 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T18:52:36.178 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ start_mirror cluster2:0 2023-12-10T18:52:36.178 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2:0 2023-12-10T18:52:36.178 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:52:36.178 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster2:0 cluster instance 2023-12-10T18:52:36.178 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster2:0 2023-12-10T18:52:36.179 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T18:52:36.179 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:52:36.179 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster2 2023-12-10T18:52:36.179 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T18:52:36.179 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster2:0 ] 2023-12-10T18:52:36.179 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster2 2023-12-10T18:52:36.179 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster2 2023-12-10T18:52:36.179 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T18:52:36.179 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T18:52:36.179 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 1 2023-12-10T18:52:36.179 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 2023-12-10T18:52:36.179 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ start_mirror cluster2:1 2023-12-10T18:52:36.179 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2:1 2023-12-10T18:52:36.179 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:52:36.179 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster2:1 cluster instance 2023-12-10T18:52:36.179 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster2:1 2023-12-10T18:52:36.180 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T18:52:36.180 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:52:36.180 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster2 2023-12-10T18:52:36.180 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T18:52:36.180 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster2:1 ] 2023-12-10T18:52:36.180 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster2 2023-12-10T18:52:36.180 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster2 2023-12-10T18:52:36.180 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T18:52:36.180 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T18:52:36.180 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 1 2023-12-10T18:52:36.180 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 2023-12-10T18:52:36.180 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ start_mirror cluster2:2 2023-12-10T18:52:36.180 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2:2 2023-12-10T18:52:36.180 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:52:36.180 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster2:2 cluster instance 2023-12-10T18:52:36.180 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster2:2 2023-12-10T18:52:36.180 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T18:52:36.181 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:52:36.181 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster2 2023-12-10T18:52:36.181 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T18:52:36.181 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster2:2 ] 2023-12-10T18:52:36.181 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster2 2023-12-10T18:52:36.181 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster2 2023-12-10T18:52:36.181 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T18:52:36.181 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T18:52:36.181 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 1 2023-12-10T18:52:36.181 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 2023-12-10T18:52:36.181 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ start_mirror cluster2:3 2023-12-10T18:52:36.181 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2:3 2023-12-10T18:52:36.181 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:52:36.181 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster2:3 cluster instance 2023-12-10T18:52:36.181 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster2:3 2023-12-10T18:52:36.181 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T18:52:36.182 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:52:36.182 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster2 2023-12-10T18:52:36.182 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T18:52:36.182 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 3 = cluster2:3 ] 2023-12-10T18:52:36.182 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster2 2023-12-10T18:52:36.182 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster2 2023-12-10T18:52:36.182 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=3 2023-12-10T18:52:36.182 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T18:52:36.182 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 1 2023-12-10T18:52:36.182 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 2023-12-10T18:52:36.182 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ testlog TEST: add image and test replay after client crashes 2023-12-10T18:52:36.182 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ tee+ -a /tmp/tmp.qNjRyMq75D/rbd-mirror.test.log 2023-12-10T18:52:36.182 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:date +%F %T 2023-12-10T18:52:36.182 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo 2023-12-10 18:52:36 TEST: add image and test replay after client crashes 2023-12-10T18:52:36.182 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:2023-12-10 18:52:36 TEST: add image and test replay after client crashes 2023-12-10T18:52:36.183 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=test 2023-12-10T18:52:36.183 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image_and_enable_mirror cluster2 mirror test snapshot 512M 2023-12-10T18:52:36.183 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T18:52:36.183 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T18:52:36.183 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:52:36.183 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T18:52:36.183 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:52:36.183 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T18:52:36.183 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T18:52:36.183 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n snapshot ] 2023-12-10T18:52:36.183 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T18:52:36.183 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image cluster2 mirror test 512M 2023-12-10T18:52:36.183 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T18:52:36.183 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T18:52:36.183 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:52:36.183 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T18:52:36.184 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:52:36.184 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T18:52:36.184 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local size=128 2023-12-10T18:52:36.184 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n 512M ] 2023-12-10T18:52:36.184 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ size=512M 2023-12-10T18:52:36.184 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T18:52:36.184 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/test 2023-12-10T18:52:36.218 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ is_pool_mirror_mode_image mirror 2023-12-10T18:52:36.219 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:52:36.219 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ image = image ] 2023-12-10T18:52:36.219 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:52:36.219 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ enable_mirror cluster2 mirror test snapshot 2023-12-10T18:52:36.219 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T18:52:36.219 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:52:36.219 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:52:36.219 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T18:52:36.219 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/test snapshot 2023-12-10T18:52:37.170 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirroring enabled 2023-12-10T18:52:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 info mirror/test 2023-12-10T18:52:37.226 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'test': 2023-12-10T18:52:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 512 MiB in 128 objects 2023-12-10T18:52:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T18:52:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 1 2023-12-10T18:52:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 10c58fa8c944 2023-12-10T18:52:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.10c58fa8c944 2023-12-10T18:52:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T18:52:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering 2023-12-10T18:52:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T18:52:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T18:52:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 18:52:36 2023 2023-12-10T18:52:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 18:52:36 2023 2023-12-10T18:52:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 18:52:36 2023 2023-12-10T18:52:37.228 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T18:52:37.228 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T18:52:37.228 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: c93b253b-04a0-4ada-b252-fcc724b6756e 2023-12-10T18:52:37.228 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: true 2023-12-10T18:52:37.231 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-10T18:52:37.231 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:52:37.231 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:52:37.231 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:52:37.231 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-10T18:52:37.231 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:52:37.231 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:52:37.231 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:52:37.231 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=started 2023-12-10T18:52:37.231 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T18:52:37.231 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T18:52:38.232 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-10T18:52:38.232 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:52:38.232 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:52:38.232 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:52:38.232 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=started 2023-12-10T18:52:38.232 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_result 2023-12-10T18:52:38.232 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=stopped 2023-12-10T18:52:38.233 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-10T18:52:38.233 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_instance=cluster1 2023-12-10T18:52:38.233 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T18:52:38.233 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:52:38.233 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance=cluster1 2023-12-10T18:52:38.233 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loop_instance 2023-12-10T18:52:38.233 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ + sleep 0 2023-12-10T18:52:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:grep -i state 2023-12-10T18:52:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T18:52:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T18:52:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-10T18:52:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T18:52:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T18:52:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:52:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T18:52:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T18:52:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T18:52:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:52:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:52:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T18:52:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T18:52:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T18:52:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:52:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T18:52:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T18:52:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T18:52:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T18:52:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:52:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:52:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T18:52:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T18:52:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T18:52:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:52:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:52:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T18:52:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T18:52:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T18:52:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T18:52:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T18:52:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T18:52:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T18:52:38.263 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T18:52:38.263 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T18:52:38.263 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T18:52:38.263 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-10T18:52:38.375 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-10T18:52:38.376 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T18:52:38.376 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T18:52:38.378 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-10T18:52:38.378 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:1 2023-12-10T18:52:38.378 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T18:52:38.378 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:52:38.378 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-10T18:52:38.378 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T18:52:38.378 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T18:52:38.378 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:52:38.378 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:52:38.378 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T18:52:38.378 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T18:52:38.378 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T18:52:38.379 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:52:38.379 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T18:52:38.379 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T18:52:38.379 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-10T18:52:38.379 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:1 2023-12-10T18:52:38.379 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:52:38.379 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:52:38.379 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-10T18:52:38.379 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T18:52:38.379 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T18:52:38.379 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:52:38.379 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:52:38.379 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T18:52:38.379 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T18:52:38.379 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T18:52:38.380 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T18:52:38.380 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T18:52:38.380 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T18:52:38.380 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-10T18:52:38.397 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T18:52:38.398 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T18:52:38.398 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T18:52:38.398 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-10T18:52:38.501 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-10T18:52:38.501 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T18:52:38.501 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T18:52:38.503 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-10T18:52:38.504 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:2 2023-12-10T18:52:38.504 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T18:52:38.504 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:52:38.504 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-10T18:52:38.504 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T18:52:38.504 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T18:52:38.504 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:52:38.504 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:52:38.504 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T18:52:38.504 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T18:52:38.504 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T18:52:38.504 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:52:38.504 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T18:52:38.504 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T18:52:38.505 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-10T18:52:38.505 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:2 2023-12-10T18:52:38.505 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:52:38.505 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:52:38.505 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-10T18:52:38.505 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T18:52:38.505 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T18:52:38.505 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:52:38.505 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:52:38.505 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T18:52:38.505 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T18:52:38.505 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T18:52:38.505 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T18:52:38.505 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T18:52:38.505 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T18:52:38.505 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-10T18:52:38.528 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T18:52:38.529 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T18:52:38.529 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T18:52:38.529 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-10T18:52:38.638 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-10T18:52:38.639 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T18:52:38.639 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T18:52:38.641 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-10T18:52:38.641 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:3 2023-12-10T18:52:38.641 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T18:52:38.641 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:52:38.641 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-10T18:52:38.641 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:3 2023-12-10T18:52:38.641 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T18:52:38.642 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:52:38.642 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:52:38.642 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T18:52:38.642 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 3 = cluster1:3 ] 2023-12-10T18:52:38.642 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T18:52:38.642 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:52:38.642 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=3 2023-12-10T18:52:38.642 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T18:52:38.642 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-10T18:52:38.642 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:3 2023-12-10T18:52:38.642 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:52:38.642 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:52:38.642 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-10T18:52:38.642 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:3 2023-12-10T18:52:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T18:52:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:52:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:52:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T18:52:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 3 = cluster1:3 ] 2023-12-10T18:52:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T18:52:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T18:52:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=3 2023-12-10T18:52:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T18:52:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-10T18:52:38.666 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T18:52:38.666 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T18:52:38.666 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T18:52:38.666 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-10T18:52:38.770 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-10T18:52:38.770 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T18:52:38.770 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T18:52:38.774 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T18:52:39.589 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.42017171558750654) > weight (0.25) 2023-12-10T18:52:39.589 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.7018100198971156) > weight (0.25) 2023-12-10T18:52:39.589 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2023-12-10T18:52:39.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2023-12-10T18:52:39.593 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2023-12-10T18:52:39.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi076.stderr:2023-12-10T18:52:39.587+0000 7fbfba534640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 39903) UID: 1000 2023-12-10T18:52:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T18:52:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T18:52:39.775 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-10T18:52:39.775 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T18:52:39.775 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T18:52:39.775 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:52:39.775 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T18:52:39.775 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T18:52:39.775 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T18:52:39.775 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:52:39.775 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:52:39.775 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T18:52:39.775 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T18:52:39.775 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T18:52:39.776 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:52:39.776 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T18:52:39.776 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T18:52:39.776 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T18:52:39.776 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T18:52:39.776 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:52:39.776 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:52:39.776 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T18:52:39.776 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T18:52:39.776 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T18:52:39.776 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:52:39.776 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:52:39.776 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T18:52:39.776 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T18:52:39.776 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T18:52:39.777 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T18:52:39.777 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T18:52:39.777 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T18:52:39.777 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T18:52:39.803 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T18:52:39.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T18:52:39.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T18:52:39.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-10T18:52:39.912 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-10T18:52:39.912 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T18:52:39.912 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T18:52:39.915 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-10T18:52:39.915 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:1 2023-12-10T18:52:39.915 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T18:52:39.915 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:52:39.915 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-10T18:52:39.915 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T18:52:39.915 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T18:52:39.915 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:52:39.915 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:52:39.915 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T18:52:39.916 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T18:52:39.916 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T18:52:39.916 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:52:39.916 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T18:52:39.916 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T18:52:39.916 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-10T18:52:39.916 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:1 2023-12-10T18:52:39.916 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:52:39.916 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:52:39.916 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-10T18:52:39.916 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T18:52:39.916 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T18:52:39.916 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:52:39.916 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:52:39.916 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T18:52:39.917 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T18:52:39.917 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T18:52:39.917 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T18:52:39.917 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T18:52:39.917 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T18:52:39.917 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-10T18:52:39.938 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T18:52:39.939 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T18:52:39.939 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T18:52:39.939 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-10T18:52:40.036 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-10T18:52:40.036 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T18:52:40.036 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T18:52:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-10T18:52:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:2 2023-12-10T18:52:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T18:52:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:52:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-10T18:52:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T18:52:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T18:52:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:52:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:52:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T18:52:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T18:52:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T18:52:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:52:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T18:52:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T18:52:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-10T18:52:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:2 2023-12-10T18:52:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:52:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:52:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-10T18:52:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T18:52:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T18:52:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:52:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:52:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T18:52:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T18:52:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T18:52:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T18:52:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T18:52:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T18:52:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-10T18:52:40.064 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T18:52:40.064 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T18:52:40.064 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T18:52:40.064 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-10T18:52:40.159 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2023-12-10T18:52:40.161 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-10T18:52:40.162 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:3 2023-12-10T18:52:40.162 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T18:52:40.162 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:52:40.162 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-10T18:52:40.162 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:3 2023-12-10T18:52:40.162 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T18:52:40.162 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:52:40.162 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:52:40.162 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T18:52:40.162 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 3 = cluster1:3 ] 2023-12-10T18:52:40.162 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T18:52:40.162 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:52:40.162 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=3 2023-12-10T18:52:40.162 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T18:52:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-10T18:52:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:3 2023-12-10T18:52:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:52:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:52:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-10T18:52:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:3 2023-12-10T18:52:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T18:52:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:52:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:52:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T18:52:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 3 = cluster1:3 ] 2023-12-10T18:52:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T18:52:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T18:52:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=3 2023-12-10T18:52:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T18:52:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-10T18:52:40.181 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T18:52:40.181 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T18:52:40.181 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T18:52:40.181 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-10T18:52:40.277 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-10T18:52:40.278 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T18:52:40.278 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T18:52:40.279 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 2 2023-12-10T18:52:42.281 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T18:52:42.281 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T18:52:42.283 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-10T18:52:42.283 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T18:52:42.283 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T18:52:42.283 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:52:42.283 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T18:52:42.283 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T18:52:42.283 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T18:52:42.283 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:52:42.284 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:52:42.284 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T18:52:42.284 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T18:52:42.284 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T18:52:42.284 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:52:42.284 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T18:52:42.284 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T18:52:42.284 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T18:52:42.284 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T18:52:42.284 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:52:42.284 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:52:42.284 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T18:52:42.284 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T18:52:42.284 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T18:52:42.284 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:52:42.284 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:52:42.285 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T18:52:42.285 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T18:52:42.285 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T18:52:42.285 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T18:52:42.285 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T18:52:42.285 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T18:52:42.285 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T18:52:42.309 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T18:52:42.309 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T18:52:42.309 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T18:52:42.309 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-10T18:52:42.411 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-10T18:52:42.411 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T18:52:42.412 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T18:52:42.414 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-10T18:52:42.414 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:1 2023-12-10T18:52:42.414 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T18:52:42.414 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:52:42.414 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-10T18:52:42.414 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T18:52:42.414 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T18:52:42.415 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:52:42.415 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:52:42.415 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T18:52:42.415 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T18:52:42.415 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T18:52:42.415 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:52:42.415 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T18:52:42.415 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T18:52:42.415 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-10T18:52:42.415 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:1 2023-12-10T18:52:42.415 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:52:42.415 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:52:42.415 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-10T18:52:42.415 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T18:52:42.415 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T18:52:42.416 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:52:42.416 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:52:42.416 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T18:52:42.416 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T18:52:42.416 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T18:52:42.416 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T18:52:42.416 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T18:52:42.416 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T18:52:42.416 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-10T18:52:42.439 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T18:52:42.439 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T18:52:42.439 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T18:52:42.439 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-10T18:52:42.553 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-10T18:52:42.554 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T18:52:42.554 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T18:52:42.556 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-10T18:52:42.556 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:2 2023-12-10T18:52:42.556 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T18:52:42.556 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:52:42.556 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-10T18:52:42.556 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T18:52:42.556 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T18:52:42.556 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:52:42.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:52:42.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T18:52:42.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T18:52:42.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T18:52:42.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:52:42.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T18:52:42.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T18:52:42.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-10T18:52:42.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:2 2023-12-10T18:52:42.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:52:42.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:52:42.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-10T18:52:42.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T18:52:42.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T18:52:42.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:52:42.558 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:52:42.558 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T18:52:42.558 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T18:52:42.558 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T18:52:42.558 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T18:52:42.558 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T18:52:42.558 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T18:52:42.558 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-10T18:52:42.581 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T18:52:42.581 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T18:52:42.581 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T18:52:42.581 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-10T18:52:42.676 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2023-12-10T18:52:42.679 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-10T18:52:42.679 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:3 2023-12-10T18:52:42.679 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T18:52:42.679 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:52:42.679 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-10T18:52:42.679 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:3 2023-12-10T18:52:42.679 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T18:52:42.679 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:52:42.679 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:52:42.679 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T18:52:42.679 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 3 = cluster1:3 ] 2023-12-10T18:52:42.679 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T18:52:42.679 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:52:42.680 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=3 2023-12-10T18:52:42.680 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T18:52:42.680 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-10T18:52:42.680 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:3 2023-12-10T18:52:42.680 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:52:42.680 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:52:42.680 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-10T18:52:42.680 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:3 2023-12-10T18:52:42.680 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T18:52:42.680 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:52:42.680 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:52:42.680 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T18:52:42.680 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 3 = cluster1:3 ] 2023-12-10T18:52:42.680 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T18:52:42.680 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T18:52:42.681 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=3 2023-12-10T18:52:42.681 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T18:52:42.681 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-10T18:52:42.704 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T18:52:42.704 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T18:52:42.704 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T18:52:42.704 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-10T18:52:42.806 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-10T18:52:42.806 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T18:52:42.806 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T18:52:42.807 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 4 2023-12-10T18:52:44.594 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2023-12-10T18:52:44.594 INFO:teuthology.orchestra.run:waiting for 600 2023-12-10T18:52:44.594 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2023-12-10T18:52:44.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2023-12-10T18:52:44.595 DEBUG:teuthology.orchestra.run.smithi076:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2023-12-10T18:52:44.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2023-12-10T18:52:44.597 INFO:tasks.rbd_mirror_thrash:waiting for 64 secs before thrashing 2023-12-10T18:52:46.589 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.3309130321191236) > weight (0.25) 2023-12-10T18:52:46.589 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2023-12-10T18:52:46.590 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2023-12-10T18:52:46.590 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2023-12-10T18:52:46.591 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr:2023-12-10T18:52:46.587+0000 7f553d377640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 39912) UID: 1000 2023-12-10T18:52:46.808 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T18:52:46.809 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T18:52:46.810 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-10T18:52:46.811 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T18:52:46.811 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T18:52:46.811 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:52:46.811 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T18:52:46.811 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T18:52:46.811 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T18:52:46.811 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:52:46.811 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:52:46.811 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T18:52:46.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T18:52:46.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T18:52:46.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:52:46.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T18:52:46.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T18:52:46.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T18:52:46.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T18:52:46.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:52:46.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:52:46.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T18:52:46.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T18:52:46.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T18:52:46.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:52:46.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:52:46.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T18:52:46.813 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T18:52:46.813 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T18:52:46.813 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T18:52:46.813 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T18:52:46.813 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T18:52:46.813 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T18:52:46.838 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T18:52:46.838 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T18:52:46.838 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T18:52:46.838 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-10T18:52:46.950 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-10T18:52:46.950 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T18:52:46.950 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T18:52:46.953 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-10T18:52:46.954 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:1 2023-12-10T18:52:46.954 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T18:52:46.954 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:52:46.954 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-10T18:52:46.954 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T18:52:46.954 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T18:52:46.954 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:52:46.954 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:52:46.954 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T18:52:46.954 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T18:52:46.954 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T18:52:46.954 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:52:46.954 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T18:52:46.955 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T18:52:46.955 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-10T18:52:46.955 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:1 2023-12-10T18:52:46.955 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:52:46.955 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:52:46.955 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-10T18:52:46.955 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T18:52:46.955 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T18:52:46.955 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:52:46.955 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:52:46.955 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T18:52:46.955 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T18:52:46.955 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T18:52:46.955 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T18:52:46.955 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T18:52:46.956 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T18:52:46.956 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-10T18:52:46.976 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T18:52:46.976 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T18:52:46.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T18:52:46.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-10T18:52:47.085 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-10T18:52:47.085 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T18:52:47.085 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T18:52:47.087 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-10T18:52:47.087 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:2 2023-12-10T18:52:47.087 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T18:52:47.087 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:52:47.087 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-10T18:52:47.087 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T18:52:47.087 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T18:52:47.087 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:52:47.087 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:52:47.088 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T18:52:47.088 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T18:52:47.088 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T18:52:47.088 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:52:47.088 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T18:52:47.088 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T18:52:47.088 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-10T18:52:47.088 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:2 2023-12-10T18:52:47.088 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:52:47.088 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:52:47.088 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-10T18:52:47.088 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T18:52:47.088 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T18:52:47.088 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:52:47.088 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:52:47.088 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T18:52:47.089 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T18:52:47.089 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T18:52:47.089 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T18:52:47.089 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T18:52:47.089 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T18:52:47.089 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-10T18:52:47.112 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T18:52:47.112 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T18:52:47.112 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T18:52:47.112 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-10T18:52:47.241 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:52:47.241 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ status_result= "state": "Replaying" 2023-12-10T18:52:47.241 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo "state": "Replaying" 2023-12-10T18:52:47.241 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep -i Replaying 2023-12-10T18:52:47.242 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "state": "Replaying" 2023-12-10T18:52:47.243 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=started 2023-12-10T18:52:47.243 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test started = started 2023-12-10T18:52:47.243 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:52:47.243 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ clean_snap_name= 2023-12-10T18:52:47.243 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 1 10 2023-12-10T18:52:47.244 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stress_write_image cluster2 mirror test 2023-12-10T18:52:47.244 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T18:52:47.244 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:52:47.244 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:52:47.245 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-10T18:52:47.246 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local duration=11 2023-12-10T18:52:47.246 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T18:52:47.246 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-10T18:52:47.397 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:100 IOs, 15111168 bytes 2023-12-10T18:52:47.570 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:200 IOs, 77036544 bytes 2023-12-10T18:52:47.719 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:300 IOs, 117096448 bytes 2023-12-10T18:52:47.806 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:400 IOs, 130880512 bytes 2023-12-10T18:52:47.993 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:500 IOs, 190004224 bytes 2023-12-10T18:52:48.128 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:600 IOs, 224505856 bytes 2023-12-10T18:52:48.277 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:700 IOs, 268021760 bytes 2023-12-10T18:52:48.470 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:800 IOs, 335188992 bytes 2023-12-10T18:52:48.563 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:900 IOs, 345728000 bytes 2023-12-10T18:52:48.590 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2023-12-10T18:52:48.591 INFO:teuthology.orchestra.run:waiting for 600 2023-12-10T18:52:48.591 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2023-12-10T18:52:48.591 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-12-10T18:52:48.591 DEBUG:teuthology.orchestra.run.smithi076:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-12-10T18:52:48.593 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-12-10T18:52:48.593 INFO:tasks.rbd_mirror_thrash:waiting for 64 secs before thrashing 2023-12-10T18:52:48.709 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1000 IOs, 384533504 bytes 2023-12-10T18:52:48.810 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1100 IOs, 398628864 bytes 2023-12-10T18:52:48.931 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1200 IOs, 423708672 bytes 2023-12-10T18:52:49.151 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1300 IOs, 451940352 bytes 2023-12-10T18:52:49.395 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1400 IOs, 535365632 bytes 2023-12-10T18:52:49.522 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1500 IOs, 566504448 bytes 2023-12-10T18:52:49.672 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1600 IOs, 606827520 bytes 2023-12-10T18:52:49.792 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1700 IOs, 631425024 bytes 2023-12-10T18:52:49.918 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1800 IOs, 652069888 bytes 2023-12-10T18:52:50.014 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1900 IOs, 672259072 bytes 2023-12-10T18:52:50.218 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2000 IOs, 747550720 bytes 2023-12-10T18:52:50.454 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2100 IOs, 822507520 bytes 2023-12-10T18:52:50.630 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2200 IOs, 869469184 bytes 2023-12-10T18:52:50.826 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2300 IOs, 937953280 bytes 2023-12-10T18:52:51.014 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2400 IOs, 990311424 bytes 2023-12-10T18:52:51.167 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2500 IOs, 1048987648 bytes 2023-12-10T18:52:51.368 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2600 IOs, 1095895040 bytes 2023-12-10T18:52:51.473 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2700 IOs, 1117889536 bytes 2023-12-10T18:52:51.607 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2800 IOs, 1144815616 bytes 2023-12-10T18:52:51.726 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2900 IOs, 1172427776 bytes 2023-12-10T18:52:51.850 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3000 IOs, 1202096128 bytes 2023-12-10T18:52:52.005 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3100 IOs, 1261409280 bytes 2023-12-10T18:52:52.155 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3200 IOs, 1301468160 bytes 2023-12-10T18:52:52.397 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3300 IOs, 1365236736 bytes 2023-12-10T18:52:52.575 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3400 IOs, 1427640320 bytes 2023-12-10T18:52:52.801 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3500 IOs, 1497719808 bytes 2023-12-10T18:52:52.938 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3600 IOs, 1534262272 bytes 2023-12-10T18:52:53.014 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3700 IOs, 1535800320 bytes 2023-12-10T18:52:53.102 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3800 IOs, 1546785792 bytes 2023-12-10T18:52:53.197 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3900 IOs, 1561407488 bytes 2023-12-10T18:52:53.296 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4000 IOs, 1580573696 bytes 2023-12-10T18:52:53.448 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4100 IOs, 1621610496 bytes 2023-12-10T18:52:53.583 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4200 IOs, 1647932416 bytes 2023-12-10T18:52:53.826 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4300 IOs, 1725225984 bytes 2023-12-10T18:52:54.065 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4400 IOs, 1808388096 bytes 2023-12-10T18:52:54.369 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4500 IOs, 1819367424 bytes 2023-12-10T18:52:54.625 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4600 IOs, 1928159232 bytes 2023-12-10T18:52:54.762 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4700 IOs, 1971016704 bytes 2023-12-10T18:52:54.949 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4800 IOs, 2011654144 bytes 2023-12-10T18:52:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4900 IOs, 2088868864 bytes 2023-12-10T18:52:55.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5000 IOs, 2131428352 bytes 2023-12-10T18:52:55.475 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5100 IOs, 2186808320 bytes 2023-12-10T18:52:55.652 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5200 IOs, 2233089024 bytes 2023-12-10T18:52:55.840 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5300 IOs, 2291720192 bytes 2023-12-10T18:52:56.025 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5400 IOs, 2338893824 bytes 2023-12-10T18:52:56.166 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5500 IOs, 2364959744 bytes 2023-12-10T18:52:56.245 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5600 IOs, 2372343808 bytes 2023-12-10T18:52:56.432 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5700 IOs, 2448977920 bytes 2023-12-10T18:52:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5800 IOs, 2480102400 bytes 2023-12-10T18:52:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5900 IOs, 2571553792 bytes 2023-12-10T18:52:57.014 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6000 IOs, 2627608576 bytes 2023-12-10T18:52:57.211 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6100 IOs, 2688403456 bytes 2023-12-10T18:52:57.306 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6200 IOs, 2702932992 bytes 2023-12-10T18:52:57.595 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6300 IOs, 2799377408 bytes 2023-12-10T18:52:57.834 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6400 IOs, 2876186624 bytes 2023-12-10T18:52:58.006 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6500 IOs, 2919546880 bytes 2023-12-10T18:52:58.181 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6600 IOs, 2966792192 bytes 2023-12-10T18:52:58.253 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=124 2023-12-10T18:52:58.253 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T18:52:58.253 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 124 -eq 124 ] 2023-12-10T18:52:58.253 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:52:58.253 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-10T18:52:58.253 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:52:58.253 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:52:58.253 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:52:58.253 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state_pattern=up+replaying 2023-12-10T18:52:58.253 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local description_pattern= 2023-12-10T18:52:58.253 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local service_pattern= 2023-12-10T18:52:58.254 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T18:52:59.255 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-10T18:52:59.255 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:52:59.255 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:52:59.255 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:52:59.255 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state_pattern=up+replaying 2023-12-10T18:52:59.255 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local description_pattern= 2023-12-10T18:52:59.255 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local service_pattern= 2023-12-10T18:52:59.255 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-test.mirror_status 2023-12-10T18:52:59.255 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-test.mirror_status 2023-12-10T18:52:59.256 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T18:52:59.256 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T18:52:59.257 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2023-12-10T18:52:59.257 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ tee /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T18:52:59.324 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:test: 2023-12-10T18:52:59.324 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: global_id: c93b253b-04a0-4ada-b252-fcc724b6756e 2023-12-10T18:52:59.324 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: state: down+unknown 2023-12-10T18:52:59.324 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: description: status not found 2023-12-10T18:52:59.324 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: last_update: 2023-12-10T18:52:59.324 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: peer_sites: 2023-12-10T18:52:59.324 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: name: cluster2 2023-12-10T18:52:59.324 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: state: up+stopped 2023-12-10T18:52:59.324 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: description: local image is primary 2023-12-10T18:52:59.324 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: last_update: 2023-12-10 18:52:50 2023-12-10T18:52:59.331 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T18:52:59.333 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 1 2023-12-10T18:52:59.333 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 2 2023-12-10T18:53:01.334 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-10T18:53:01.335 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:53:01.335 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:53:01.335 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:53:01.335 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state_pattern=up+replaying 2023-12-10T18:53:01.335 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local description_pattern= 2023-12-10T18:53:01.335 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local service_pattern= 2023-12-10T18:53:01.335 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-test.mirror_status 2023-12-10T18:53:01.335 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-test.mirror_status 2023-12-10T18:53:01.335 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T18:53:01.336 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T18:53:01.337 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2023-12-10T18:53:01.337 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ tee /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T18:53:01.400 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:test: 2023-12-10T18:53:01.401 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: global_id: c93b253b-04a0-4ada-b252-fcc724b6756e 2023-12-10T18:53:01.401 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: state: down+unknown 2023-12-10T18:53:01.401 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: description: status not found 2023-12-10T18:53:01.401 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: last_update: 2023-12-10T18:53:01.401 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: peer_sites: 2023-12-10T18:53:01.401 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: name: cluster2 2023-12-10T18:53:01.401 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: state: up+stopped 2023-12-10T18:53:01.401 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: description: local image is primary 2023-12-10T18:53:01.401 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: last_update: 2023-12-10 18:52:50 2023-12-10T18:53:01.408 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T18:53:01.409 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 1 2023-12-10T18:53:01.410 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 4 2023-12-10T18:53:04.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr:2023-12-10T18:53:04.958+0000 7f738d1b2640 -1 rbd::mirror::PoolReplayer: 0x562139a8b500 handle_remote_pool_meta_updated: remote pool metadata updated unexpectedly 2023-12-10T18:53:04.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi076.stderr:2023-12-10T18:53:04.970+0000 7f579eca1640 -1 rbd::mirror::PoolReplayer: 0x561390afb500 handle_remote_pool_meta_updated: remote pool metadata updated unexpectedly 2023-12-10T18:53:04.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi076.stderr:2023-12-10T18:53:04.974+0000 7f430e147640 -1 rbd::mirror::PoolReplayer: 0x55af1ff5b500 handle_remote_pool_meta_updated: remote pool metadata updated unexpectedly 2023-12-10T18:53:05.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr:2023-12-10T18:53:04.998+0000 7f73aaa2c880 -1 rbd::mirror::Mirror: 0x562138dcbd40 update_pool_replayers: restarting failed pool replayer for uuid: cea0e0df-3ba5-45fb-ab55-c5b809a43d2b cluster: cluster2 client: client.mirror 2023-12-10T18:53:05.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi076.stderr:2023-12-10T18:53:05.014+0000 7f57bc51b880 -1 rbd::mirror::Mirror: 0x56138f1add40 update_pool_replayers: restarting failed pool replayer for uuid: cea0e0df-3ba5-45fb-ab55-c5b809a43d2b cluster: cluster2 client: client.mirror 2023-12-10T18:53:05.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi076.stderr:2023-12-10T18:53:05.022+0000 7f432b9c1880 -1 rbd::mirror::Mirror: 0x55af1e5f9d40 update_pool_replayers: restarting failed pool replayer for uuid: cea0e0df-3ba5-45fb-ab55-c5b809a43d2b cluster: cluster2 client: client.mirror 2023-12-10T18:53:05.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr:2023-12-10T18:53:05.082+0000 7f73aaa2c880 -1 rbd::mirror::Mirror: 0x562138dcbd40 update_pool_replayers: restarting failed pool replayer for uuid: 0f54ea86-1117-4a80-9b51-0eebdfe1047a cluster: cluster2 client: client.mirror 2023-12-10T18:53:05.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi076.stderr:2023-12-10T18:53:05.102+0000 7f432b9c1880 -1 rbd::mirror::Mirror: 0x55af1e5f9d40 update_pool_replayers: restarting failed pool replayer for uuid: 0f54ea86-1117-4a80-9b51-0eebdfe1047a cluster: cluster2 client: client.mirror 2023-12-10T18:53:05.116 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi076.stderr:2023-12-10T18:53:05.110+0000 7f57bc51b880 -1 rbd::mirror::Mirror: 0x56138f1add40 update_pool_replayers: restarting failed pool replayer for uuid: 0f54ea86-1117-4a80-9b51-0eebdfe1047a cluster: cluster2 client: client.mirror 2023-12-10T18:53:05.411 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-10T18:53:05.412 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:53:05.412 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:53:05.412 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:53:05.412 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state_pattern=up+replaying 2023-12-10T18:53:05.412 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local description_pattern= 2023-12-10T18:53:05.412 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local service_pattern= 2023-12-10T18:53:05.412 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-test.mirror_status 2023-12-10T18:53:05.412 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-test.mirror_status 2023-12-10T18:53:05.412 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T18:53:05.414 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T18:53:05.414 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2023-12-10T18:53:05.417 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ tee /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T18:53:05.481 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:test: 2023-12-10T18:53:05.482 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: global_id: c93b253b-04a0-4ada-b252-fcc724b6756e 2023-12-10T18:53:05.482 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: state: down+unknown 2023-12-10T18:53:05.482 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: description: status not found 2023-12-10T18:53:05.482 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: last_update: 2023-12-10T18:53:05.482 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: peer_sites: 2023-12-10T18:53:05.482 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: name: cluster2 2023-12-10T18:53:05.482 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: state: up+stopped 2023-12-10T18:53:05.482 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: description: local image is primary 2023-12-10T18:53:05.482 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: last_update: 2023-12-10 18:52:50 2023-12-10T18:53:05.491 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T18:53:05.492 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 1 2023-12-10T18:53:05.492 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 8 2023-12-10T18:53:13.494 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-10T18:53:13.494 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:53:13.495 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:53:13.495 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:53:13.495 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state_pattern=up+replaying 2023-12-10T18:53:13.495 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local description_pattern= 2023-12-10T18:53:13.495 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local service_pattern= 2023-12-10T18:53:13.495 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-test.mirror_status 2023-12-10T18:53:13.495 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-test.mirror_status 2023-12-10T18:53:13.495 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T18:53:13.496 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T18:53:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2023-12-10T18:53:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ tee /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T18:53:13.568 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:test: 2023-12-10T18:53:13.568 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: global_id: c93b253b-04a0-4ada-b252-fcc724b6756e 2023-12-10T18:53:13.568 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: state: down+unknown 2023-12-10T18:53:13.569 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: description: status not found 2023-12-10T18:53:13.569 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: last_update: 2023-12-10T18:53:13.569 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: peer_sites: 2023-12-10T18:53:13.569 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: name: cluster2 2023-12-10T18:53:13.569 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: state: up+stopped 2023-12-10T18:53:13.569 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: description: local image is primary 2023-12-10T18:53:13.569 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: last_update: 2023-12-10 18:52:50 2023-12-10T18:53:13.577 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T18:53:13.578 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 1 2023-12-10T18:53:13.578 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 8 2023-12-10T18:53:21.581 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-10T18:53:21.581 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:53:21.581 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:53:21.582 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:53:21.582 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state_pattern=up+replaying 2023-12-10T18:53:21.582 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local description_pattern= 2023-12-10T18:53:21.582 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local service_pattern= 2023-12-10T18:53:21.582 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-test.mirror_status 2023-12-10T18:53:21.582 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-test.mirror_status 2023-12-10T18:53:21.583 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T18:53:21.584 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T18:53:21.584 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ CEPH_ARGS= rbd --cluster+ cluster1 mirror image status mirror/test 2023-12-10T18:53:21.584 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:tee /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T18:53:21.656 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:test: 2023-12-10T18:53:21.656 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: global_id: c93b253b-04a0-4ada-b252-fcc724b6756e 2023-12-10T18:53:21.656 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: state: up+replaying 2023-12-10T18:53:21.656 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1702234357,"remote_snapshot_timestamp":1702234357,"replay_state":"idle"} 2023-12-10T18:53:21.656 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: service: mirror.2 on smithi076 2023-12-10T18:53:21.656 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: last_update: 2023-12-10 18:53:14 2023-12-10T18:53:21.656 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: peer_sites: 2023-12-10T18:53:21.656 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: name: cluster2 2023-12-10T18:53:21.656 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: state: up+stopped 2023-12-10T18:53:21.656 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: description: local image is primary 2023-12-10T18:53:21.656 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: last_update: 2023-12-10 18:53:18 2023-12-10T18:53:21.664 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T18:53:21.665 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: state: up+replaying 2023-12-10T18:53:21.666 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep ^ description: .* /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T18:53:21.667 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1702234357,"remote_snapshot_timestamp":1702234357,"replay_state":"idle"} 2023-12-10T18:53:21.667 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T18:53:21.667 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo up+replaying 2023-12-10T18:53:21.667 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep ^up+ 2023-12-10T18:53:21.668 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:up+replaying 2023-12-10T18:53:21.669 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep service: *mirror..* on /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T18:53:21.669 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: service: mirror.2 on smithi076 2023-12-10T18:53:21.670 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T18:53:21.671 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local last_update=2023-12-10 18:53:14 2023-12-10T18:53:21.671 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying 2023-12-10 18:53:14 2023-12-10T18:53:21.671 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:53:21.671 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:53:21.672 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:53:21.672 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state_pattern=up+replaying 2023-12-10T18:53:21.672 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local prev_last_update=2023-12-10 18:53:14 2023-12-10T18:53:21.672 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror.mirror_status 2023-12-10T18:53:21.672 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror.mirror_status 2023-12-10T18:53:21.672 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T18:53:21.672 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror.mirror_status 2023-12-10T18:53:21.672 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-10T18:53:21.716 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: failed to query services: (13) Permission denied 2023-12-10T18:53:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local last_update state 2023-12-10T18:53:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2023-12-10T18:53:21.760 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ last_update=2023-12-10 18:53:14 2023-12-10T18:53:21.760 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2023-12-10T18:53:21.763 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ state=up+replaying 2023-12-10T18:53:21.763 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo up+replaying 2023-12-10T18:53:21.763 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep up+replaying 2023-12-10T18:53:21.764 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:up+replaying 2023-12-10T18:53:21.765 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:53:21.765 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:53:21.765 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ snap_name=snap1 2023-12-10T18:53:21.765 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror test snap1 2023-12-10T18:53:21.765 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T18:53:21.765 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:53:21.765 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:53:21.765 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap1 2023-12-10T18:53:21.765 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1 --debug-rbd=20 --debug-journaler=20 2023-12-10T18:53:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-10T18:53:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:53:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:53:28.657 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:53:28.657 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-10T18:53:28.657 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:53:28.657 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:53:28.657 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:53:28.657 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=started 2023-12-10T18:53:28.657 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T18:53:28.657 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T18:53:29.658 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-10T18:53:29.658 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:53:29.658 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:53:29.658 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:53:29.658 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=started 2023-12-10T18:53:29.658 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_result 2023-12-10T18:53:29.658 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=stopped 2023-12-10T18:53:29.658 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ + admin_daemons cluster1 rbd mirror status mirror/test 2023-12-10T18:53:29.658 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:grep+ -ilocal state cluster_instance=cluster1 2023-12-10T18:53:29.658 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T18:53:29.658 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T18:53:29.658 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:53:29.659 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance=cluster1 2023-12-10T18:53:29.659 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loop_instance 2023-12-10T18:53:29.659 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T18:53:29.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T18:53:29.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T18:53:29.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-10T18:53:29.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T18:53:29.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T18:53:29.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:53:29.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T18:53:29.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T18:53:29.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T18:53:29.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:53:29.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:53:29.661 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T18:53:29.661 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T18:53:29.661 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T18:53:29.661 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:53:29.661 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T18:53:29.661 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T18:53:29.661 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T18:53:29.661 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T18:53:29.661 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:53:29.661 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:53:29.661 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T18:53:29.661 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T18:53:29.661 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T18:53:29.661 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:53:29.661 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:53:29.661 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T18:53:29.662 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T18:53:29.662 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T18:53:29.662 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T18:53:29.662 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T18:53:29.662 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T18:53:29.662 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T18:53:29.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T18:53:29.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T18:53:29.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T18:53:29.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-10T18:53:29.792 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-10T18:53:29.793 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T18:53:29.793 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T18:53:29.798 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-10T18:53:29.798 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:1 2023-12-10T18:53:29.798 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T18:53:29.798 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:53:29.798 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-10T18:53:29.798 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T18:53:29.798 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T18:53:29.798 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:53:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:53:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T18:53:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T18:53:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T18:53:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:53:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T18:53:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T18:53:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-10T18:53:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:1 2023-12-10T18:53:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:53:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:53:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-10T18:53:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T18:53:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T18:53:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:53:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:53:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T18:53:29.800 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T18:53:29.800 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T18:53:29.800 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T18:53:29.800 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T18:53:29.800 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T18:53:29.800 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-10T18:53:29.814 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T18:53:29.814 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T18:53:29.814 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T18:53:29.814 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-10T18:53:29.919 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-10T18:53:29.919 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T18:53:29.919 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T18:53:29.921 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-10T18:53:29.921 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:2 2023-12-10T18:53:29.921 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T18:53:29.921 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:53:29.921 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-10T18:53:29.922 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T18:53:29.922 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T18:53:29.922 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:53:29.922 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:53:29.922 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T18:53:29.922 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T18:53:29.922 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T18:53:29.922 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:53:29.922 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T18:53:29.922 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T18:53:29.922 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-10T18:53:29.922 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:2 2023-12-10T18:53:29.922 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:53:29.922 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:53:29.923 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-10T18:53:29.923 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T18:53:29.923 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T18:53:29.923 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:53:29.923 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:53:29.923 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T18:53:29.923 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T18:53:29.923 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T18:53:29.923 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T18:53:29.923 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T18:53:29.923 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T18:53:29.923 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-10T18:53:29.945 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T18:53:29.945 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T18:53:29.945 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T18:53:29.945 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-10T18:53:30.073 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:53:30.074 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ status_result= "state": "Replaying" 2023-12-10T18:53:30.074 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo "state": "Replaying" 2023-12-10T18:53:30.074 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep -i Replaying 2023-12-10T18:53:30.075 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "state": "Replaying" 2023-12-10T18:53:30.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=started 2023-12-10T18:53:30.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test started = started 2023-12-10T18:53:30.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:53:30.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-10T18:53:30.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T18:53:30.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T18:53:30.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:53:30.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:53:30.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = journal ] 2023-12-10T18:53:30.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = snapshot ] 2023-12-10T18:53:30.078 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-10T18:53:30.078 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T18:53:30.078 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T18:53:30.078 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:53:30.078 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:53:30.078 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster2-mirror-test.status 2023-12-10T18:53:30.078 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2-mirror-test.status 2023-12-10T18:53:30.078 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T18:53:30.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-test.status 2023-12-10T18:53:30.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-test.status 2023-12-10T18:53:30.080 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-test.status 2023-12-10T18:53:30.080 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T18:53:30.080 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:53:30.080 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-10T18:53:30.080 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T18:53:30.080 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:53:30.080 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:53:30.080 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-10T18:53:30.679 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Snapshot ID: 6 2023-12-10T18:53:30.689 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.status 2023-12-10T18:53:30.689 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T18:53:30.689 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:53:30.690 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:53:30.690 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-test.status 2023-12-10T18:53:30.690 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-10T18:53:30.690 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T18:53:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-10T18:53:30.747 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snapshot_id=6 2023-12-10T18:53:30.747 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T18:53:30.747 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.2 2023-12-10T18:53:30.948 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:53:30.949 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:53:30.949 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:53:30.949 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:53:30.949 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:53:30.949 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-10T18:53:30.949 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T18:53:31.034 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T18:53:31.034 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T18:53:31.036 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T18:53:31.036 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T18:53:31.036 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T18:53:31.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T18:53:31.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 6 = 2023-12-10T18:53:31.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.4 2023-12-10T18:53:31.439 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:53:31.439 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:53:31.439 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:53:31.439 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:53:31.439 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:53:31.439 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-10T18:53:31.439 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T18:53:31.505 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T18:53:31.505 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T18:53:31.507 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T18:53:31.507 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T18:53:31.507 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T18:53:31.508 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T18:53:31.508 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 6 = 2023-12-10T18:53:31.508 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.8 2023-12-10T18:53:32.309 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:53:32.309 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:53:32.309 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:53:32.309 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:53:32.309 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:53:32.309 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-10T18:53:32.310 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T18:53:32.414 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T18:53:32.414 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T18:53:32.416 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T18:53:32.416 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T18:53:32.417 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T18:53:32.417 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T18:53:32.417 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 6 = 2023-12-10T18:53:32.417 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1.6 2023-12-10T18:53:34.017 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:53:34.018 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:53:34.018 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:53:34.018 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:53:34.018 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:53:34.018 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-10T18:53:34.019 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T18:53:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T18:53:34.096 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=6 2023-12-10T18:53:34.096 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 6 = 6 2023-12-10T18:53:34.096 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:53:34.096 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snap_present cluster1 mirror test snap1 2023-12-10T18:53:34.096 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:53:34.096 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:53:34.096 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:53:34.096 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap1 2023-12-10T18:53:34.096 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T18:53:34.096 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T18:53:35.097 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 info mirror/test@snap1 2023-12-10T18:53:35.161 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'test': 2023-12-10T18:53:35.161 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 512 MiB in 128 objects 2023-12-10T18:53:35.161 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T18:53:35.161 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 2 2023-12-10T18:53:35.161 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 10c0be4f5681 2023-12-10T18:53:35.161 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.10c0be4f5681 2023-12-10T18:53:35.161 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T18:53:35.161 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering, non-primary 2023-12-10T18:53:35.161 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T18:53:35.161 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T18:53:35.161 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 18:52:46 2023 2023-12-10T18:53:35.161 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 18:52:46 2023 2023-12-10T18:53:35.162 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 18:52:46 2023 2023-12-10T18:53:35.162 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: protected: False 2023-12-10T18:53:35.162 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T18:53:35.162 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T18:53:35.162 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: c93b253b-04a0-4ada-b252-fcc724b6756e 2023-12-10T18:53:35.162 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: false 2023-12-10T18:53:35.165 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:53:35.165 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T18:53:35.165 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror test snap1 2023-12-10T18:53:35.165 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:53:35.165 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:53:35.165 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap1 2023-12-10T18:53:35.165 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T18:53:35.166 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T18:53:35.166 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:53:35.166 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:53:35.166 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T18:53:35.833 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T18:53:35.848 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:53:36.864 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T18:53:36.879 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:53:37.158 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:53:37.293 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:53:37.293 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ clean_snap_name=snap1-clean 2023-12-10T18:53:37.293 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror test snap1-clean 2023-12-10T18:53:37.293 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T18:53:37.294 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:53:37.294 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:53:37.294 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap1-clean 2023-12-10T18:53:37.294 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1-clean --debug-rbd=20 --debug-journaler=20 2023-12-10T18:53:37.602 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stress_write_image cluster2 mirror test 2023-12-10T18:53:37.602 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T18:53:37.602 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:53:37.602 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:53:37.602 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-10T18:53:37.604 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local duration=5 2023-12-10T18:53:37.604 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T18:53:37.604 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-10T18:53:37.839 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:100 IOs, 25368576 bytes 2023-12-10T18:53:37.986 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:200 IOs, 48008192 bytes 2023-12-10T18:53:38.149 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:300 IOs, 82531328 bytes 2023-12-10T18:53:38.249 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:400 IOs, 117074944 bytes 2023-12-10T18:53:38.535 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:500 IOs, 202750976 bytes 2023-12-10T18:53:38.716 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:600 IOs, 258309120 bytes 2023-12-10T18:53:38.864 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:700 IOs, 309208064 bytes 2023-12-10T18:53:38.943 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:800 IOs, 313637888 bytes 2023-12-10T18:53:39.138 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:900 IOs, 391976960 bytes 2023-12-10T18:53:39.346 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1000 IOs, 443800576 bytes 2023-12-10T18:53:39.729 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1100 IOs, 512327680 bytes 2023-12-10T18:53:39.964 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1200 IOs, 595096576 bytes 2023-12-10T18:53:40.186 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1300 IOs, 675406848 bytes 2023-12-10T18:53:40.309 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1400 IOs, 697223168 bytes 2023-12-10T18:53:40.510 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1500 IOs, 761935872 bytes 2023-12-10T18:53:40.604 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1600 IOs, 772700160 bytes 2023-12-10T18:53:40.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1700 IOs, 834746368 bytes 2023-12-10T18:53:41.212 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1800 IOs, 883585024 bytes 2023-12-10T18:53:41.389 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1900 IOs, 929859584 bytes 2023-12-10T18:53:41.544 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2000 IOs, 986542080 bytes 2023-12-10T18:53:41.682 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2100 IOs, 1001155584 bytes 2023-12-10T18:53:41.793 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2200 IOs, 1022769152 bytes 2023-12-10T18:53:41.875 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2300 IOs, 1023385600 bytes 2023-12-10T18:53:41.995 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2400 IOs, 1046116352 bytes 2023-12-10T18:53:42.174 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2500 IOs, 1120861184 bytes 2023-12-10T18:53:42.345 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2600 IOs, 1153853440 bytes 2023-12-10T18:53:42.491 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2700 IOs, 1187445760 bytes 2023-12-10T18:53:42.608 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=124 2023-12-10T18:53:42.609 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T18:53:42.609 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 124 -eq 124 ] 2023-12-10T18:53:42.609 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:53:42.609 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-10T18:53:42.609 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:53:42.609 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:53:42.609 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:53:42.609 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state_pattern=up+replaying 2023-12-10T18:53:42.609 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local description_pattern= 2023-12-10T18:53:42.609 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local service_pattern= 2023-12-10T18:53:42.609 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T18:53:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-10T18:53:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:53:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:53:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:53:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state_pattern=up+replaying 2023-12-10T18:53:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local description_pattern= 2023-12-10T18:53:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local service_pattern= 2023-12-10T18:53:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-test.mirror_status 2023-12-10T18:53:43.611 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-test.mirror_status 2023-12-10T18:53:43.611 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T18:53:43.613 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T18:53:43.613 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2023-12-10T18:53:43.613 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ tee /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T18:53:43.684 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:test: 2023-12-10T18:53:43.685 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: global_id: c93b253b-04a0-4ada-b252-fcc724b6756e 2023-12-10T18:53:43.685 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: state: up+replaying 2023-12-10T18:53:43.685 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1702234357,"remote_snapshot_timestamp":1702234357,"replay_state":"idle"} 2023-12-10T18:53:43.685 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: service: mirror.2 on smithi076 2023-12-10T18:53:43.685 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: last_update: 2023-12-10 18:53:14 2023-12-10T18:53:43.685 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: peer_sites: 2023-12-10T18:53:43.685 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: name: cluster2 2023-12-10T18:53:43.685 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: state: up+stopped 2023-12-10T18:53:43.685 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: description: local image is primary 2023-12-10T18:53:43.685 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: last_update: 2023-12-10 18:53:18 2023-12-10T18:53:43.694 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T18:53:43.695 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: state: up+replaying 2023-12-10T18:53:43.695 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep ^ description: .* /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T18:53:43.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1702234357,"remote_snapshot_timestamp":1702234357,"replay_state":"idle"} 2023-12-10T18:53:43.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T18:53:43.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ + echo up+replaying 2023-12-10T18:53:43.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:grep ^up+ 2023-12-10T18:53:43.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:up+replaying 2023-12-10T18:53:43.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep service: *mirror..* on /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T18:53:43.699 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: service: mirror.2 on smithi076 2023-12-10T18:53:43.699 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T18:53:43.701 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local last_update=2023-12-10 18:53:14 2023-12-10T18:53:43.701 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying 2023-12-10 18:53:14 2023-12-10T18:53:43.701 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:53:43.701 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:53:43.701 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:53:43.701 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state_pattern=up+replaying 2023-12-10T18:53:43.701 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local prev_last_update=2023-12-10 18:53:14 2023-12-10T18:53:43.701 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror.mirror_status 2023-12-10T18:53:43.701 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror.mirror_status 2023-12-10T18:53:43.701 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T18:53:43.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror.mirror_status 2023-12-10T18:53:43.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-10T18:53:43.750 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: failed to query services: (13) Permission denied 2023-12-10T18:53:43.795 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local last_update state 2023-12-10T18:53:43.795 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2023-12-10T18:53:43.797 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ last_update=2023-12-10 18:53:14 2023-12-10T18:53:43.797 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2023-12-10T18:53:43.800 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ state=up+replaying 2023-12-10T18:53:43.800 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo up+replaying 2023-12-10T18:53:43.800 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep up+replaying 2023-12-10T18:53:43.801 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:up+replaying 2023-12-10T18:53:43.801 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:53:43.801 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:53:43.801 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ snap_name=snap2 2023-12-10T18:53:43.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror test snap2 2023-12-10T18:53:43.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T18:53:43.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:53:43.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:53:43.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap2 2023-12-10T18:53:43.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2 --debug-rbd=20 --debug-journaler=20 2023-12-10T18:53:48.598 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-12-10T18:53:48.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-12-10T18:53:48.599 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2023-12-10T18:53:48.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr:2023-12-10T18:53:48.593+0000 7f73a79e7640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 39899) UID: 1000 2023-12-10T18:53:49.600 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-12-10T18:53:49.600 INFO:teuthology.orchestra.run:waiting for 600 2023-12-10T18:53:49.600 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-12-10T18:53:49.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-10T18:53:49.601 DEBUG:teuthology.orchestra.run.smithi076:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-10T18:53:49.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-10T18:53:49.604 INFO:tasks.rbd_mirror_thrash:waiting for 78 secs before thrashing 2023-12-10T18:53:52.594 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.34163344870076207) > weight (0.25) 2023-12-10T18:53:52.594 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.8357688579430882) > weight (0.25) 2023-12-10T18:53:52.594 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2023-12-10T18:53:52.595 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2023-12-10T18:53:52.595 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2023-12-10T18:53:52.595 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2023-12-10T18:53:52.595 INFO:teuthology.orchestra.run:waiting for 600 2023-12-10T18:53:52.596 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi076.stderr:2023-12-10T18:53:52.589+0000 7f4685d1f640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 39915) UID: 1000 2023-12-10T18:53:58.697 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2023-12-10T18:53:58.697 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2023-12-10T18:53:58.697 DEBUG:teuthology.orchestra.run.smithi076:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2023-12-10T18:53:58.700 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2023-12-10T18:53:58.700 INFO:tasks.rbd_mirror_thrash:waiting for 79 secs before thrashing 2023-12-10T18:54:11.626 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-10T18:54:11.627 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:54:11.627 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:54:11.627 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:54:11.627 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-10T18:54:11.627 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:54:11.627 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:54:11.627 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:54:11.627 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=started 2023-12-10T18:54:11.627 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T18:54:11.627 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T18:54:12.627 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-10T18:54:12.628 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:54:12.628 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:54:12.628 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:54:12.628 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=started 2023-12-10T18:54:12.628 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_result 2023-12-10T18:54:12.628 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=stopped 2023-12-10T18:54:12.629 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-10T18:54:12.629 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_instance=cluster1 2023-12-10T18:54:12.629 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T18:54:12.629 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:54:12.629 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance=cluster1 2023-12-10T18:54:12.629 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loop_instance 2023-12-10T18:54:12.629 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T18:54:12.629 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep -i state 2023-12-10T18:54:12.631 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T18:54:12.631 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T18:54:12.631 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-10T18:54:12.631 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T18:54:12.631 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T18:54:12.631 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:54:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T18:54:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T18:54:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T18:54:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:54:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:54:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T18:54:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T18:54:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T18:54:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:54:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T18:54:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T18:54:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T18:54:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T18:54:12.633 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:54:12.633 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:54:12.633 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T18:54:12.633 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T18:54:12.633 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T18:54:12.633 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:54:12.633 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:54:12.633 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T18:54:12.633 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T18:54:12.633 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T18:54:12.633 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T18:54:12.633 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T18:54:12.633 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T18:54:12.633 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T18:54:12.653 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T18:54:12.653 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T18:54:12.653 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T18:54:12.653 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-10T18:54:12.757 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-10T18:54:12.758 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T18:54:12.758 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T18:54:12.760 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-10T18:54:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:1 2023-12-10T18:54:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T18:54:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:54:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-10T18:54:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T18:54:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T18:54:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:54:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:54:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T18:54:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T18:54:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T18:54:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:54:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T18:54:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T18:54:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-10T18:54:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:1 2023-12-10T18:54:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:54:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:54:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-10T18:54:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T18:54:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T18:54:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:54:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:54:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T18:54:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T18:54:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T18:54:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T18:54:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T18:54:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T18:54:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-10T18:54:12.780 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T18:54:12.780 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T18:54:12.780 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T18:54:12.780 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-10T18:54:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-10T18:54:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T18:54:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T18:54:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-10T18:54:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:2 2023-12-10T18:54:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T18:54:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:54:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-10T18:54:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T18:54:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T18:54:12.887 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:54:12.887 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:54:12.887 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T18:54:12.887 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T18:54:12.887 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T18:54:12.887 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:54:12.887 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T18:54:12.887 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T18:54:12.887 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-10T18:54:12.887 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:2 2023-12-10T18:54:12.887 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:54:12.887 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:54:12.887 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-10T18:54:12.887 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T18:54:12.887 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T18:54:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:54:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:54:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T18:54:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T18:54:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T18:54:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T18:54:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T18:54:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T18:54:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-10T18:54:12.905 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T18:54:12.905 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T18:54:12.905 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T18:54:12.905 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-10T18:54:13.033 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:54:13.033 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ status_result= "state": "Replaying" 2023-12-10T18:54:13.033 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo "state": "Replaying" 2023-12-10T18:54:13.033 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep -i Replaying 2023-12-10T18:54:13.034 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "state": "Replaying" 2023-12-10T18:54:13.035 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=started 2023-12-10T18:54:13.035 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test started = started 2023-12-10T18:54:13.036 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:54:13.036 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-10T18:54:13.036 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T18:54:13.036 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T18:54:13.036 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:54:13.036 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:54:13.036 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = journal ] 2023-12-10T18:54:13.036 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = snapshot ] 2023-12-10T18:54:13.036 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-10T18:54:13.036 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T18:54:13.036 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T18:54:13.036 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:54:13.036 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:54:13.036 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster2-mirror-test.status 2023-12-10T18:54:13.036 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2-mirror-test.status 2023-12-10T18:54:13.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T18:54:13.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-test.status 2023-12-10T18:54:13.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-test.status 2023-12-10T18:54:13.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-test.status 2023-12-10T18:54:13.038 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T18:54:13.038 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:54:13.039 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-10T18:54:13.039 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T18:54:13.039 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:54:13.039 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:54:13.039 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-10T18:54:14.653 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Snapshot ID: 9 2023-12-10T18:54:14.664 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.status 2023-12-10T18:54:14.664 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T18:54:14.664 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:54:14.664 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:54:14.664 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-test.status 2023-12-10T18:54:14.664 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-10T18:54:14.664 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T18:54:14.717 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-10T18:54:14.719 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snapshot_id=9 2023-12-10T18:54:14.719 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T18:54:14.719 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.2 2023-12-10T18:54:14.922 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:54:14.923 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:54:14.923 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:54:14.923 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:54:14.923 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:54:14.923 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-10T18:54:14.923 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T18:54:14.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T18:54:14.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T18:54:14.992 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T18:54:14.992 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T18:54:14.992 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T18:54:14.992 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T18:54:14.992 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 9 = 2023-12-10T18:54:14.992 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.4 2023-12-10T18:54:15.393 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:54:15.393 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:54:15.393 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:54:15.394 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:54:15.394 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:54:15.394 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-10T18:54:15.394 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T18:54:15.462 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T18:54:15.462 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T18:54:15.464 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T18:54:15.464 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T18:54:15.464 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T18:54:15.464 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T18:54:15.465 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 9 = 2023-12-10T18:54:15.465 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.8 2023-12-10T18:54:16.265 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:54:16.265 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:54:16.265 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:54:16.266 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:54:16.266 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:54:16.266 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-10T18:54:16.266 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T18:54:16.336 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T18:54:16.336 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T18:54:16.338 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T18:54:16.338 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T18:54:16.338 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T18:54:16.339 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T18:54:16.339 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 9 = 2023-12-10T18:54:16.339 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1.6 2023-12-10T18:54:17.940 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:54:17.940 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:54:17.940 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:54:17.941 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:54:17.941 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:54:17.941 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-10T18:54:17.941 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T18:54:18.017 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T18:54:18.020 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=9 2023-12-10T18:54:18.021 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 9 = 9 2023-12-10T18:54:18.021 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:54:18.021 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snap_present cluster1 mirror test snap2 2023-12-10T18:54:18.021 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:54:18.021 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:54:18.021 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:54:18.021 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap2 2023-12-10T18:54:18.021 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T18:54:18.021 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T18:54:19.021 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 info mirror/test@snap2 2023-12-10T18:54:19.082 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'test': 2023-12-10T18:54:19.082 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 512 MiB in 128 objects 2023-12-10T18:54:19.083 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T18:54:19.083 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 4 2023-12-10T18:54:19.083 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 10c0be4f5681 2023-12-10T18:54:19.083 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.10c0be4f5681 2023-12-10T18:54:19.083 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T18:54:19.083 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering, non-primary 2023-12-10T18:54:19.083 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T18:54:19.083 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T18:54:19.083 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 18:52:46 2023 2023-12-10T18:54:19.083 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 18:52:46 2023 2023-12-10T18:54:19.083 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 18:52:46 2023 2023-12-10T18:54:19.084 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: protected: False 2023-12-10T18:54:19.084 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T18:54:19.084 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T18:54:19.084 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: c93b253b-04a0-4ada-b252-fcc724b6756e 2023-12-10T18:54:19.084 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: false 2023-12-10T18:54:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:54:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n snap1-clean ] 2023-12-10T18:54:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror test snap1-clean 2023-12-10T18:54:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:54:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:54:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap1-clean 2023-12-10T18:54:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T18:54:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T18:54:19.088 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:54:19.088 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:54:19.088 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T18:54:19.797 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T18:54:19.807 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:54:20.943 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T18:54:20.971 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:54:21.250 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:54:21.386 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:54:21.386 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror test snap2 2023-12-10T18:54:21.386 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:54:21.386 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:54:21.386 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap2 2023-12-10T18:54:21.386 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T18:54:21.386 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T18:54:21.386 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:54:21.387 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:54:21.387 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T18:54:22.030 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T18:54:22.046 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:54:23.192 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T18:54:23.208 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:54:23.488 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:54:23.626 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:54:23.627 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ clean_snap_name=snap2-clean 2023-12-10T18:54:23.627 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror test snap2-clean 2023-12-10T18:54:23.627 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T18:54:23.627 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:54:23.627 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:54:23.627 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap2-clean 2023-12-10T18:54:23.627 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2-clean --debug-rbd=20 --debug-journaler=20 2023-12-10T18:54:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stress_write_image cluster2 mirror test 2023-12-10T18:54:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T18:54:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:54:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:54:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-10T18:54:24.671 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local duration=11 2023-12-10T18:54:24.671 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T18:54:24.671 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-10T18:54:25.038 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:100 IOs, 86831104 bytes 2023-12-10T18:54:25.201 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:200 IOs, 123030528 bytes 2023-12-10T18:54:25.496 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:300 IOs, 224187392 bytes 2023-12-10T18:54:25.623 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:400 IOs, 250712064 bytes 2023-12-10T18:54:25.700 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:500 IOs, 259494912 bytes 2023-12-10T18:54:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:600 IOs, 313379840 bytes 2023-12-10T18:54:26.034 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:700 IOs, 347656192 bytes 2023-12-10T18:54:26.159 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:800 IOs, 372837376 bytes 2023-12-10T18:54:26.227 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:900 IOs, 373448704 bytes 2023-12-10T18:54:26.389 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1000 IOs, 431453184 bytes 2023-12-10T18:54:26.489 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1100 IOs, 440310784 bytes 2023-12-10T18:54:26.630 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1200 IOs, 496087040 bytes 2023-12-10T18:54:26.794 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1300 IOs, 531664896 bytes 2023-12-10T18:54:26.947 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1400 IOs, 576059392 bytes 2023-12-10T18:54:27.068 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1500 IOs, 595585024 bytes 2023-12-10T18:54:27.133 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1600 IOs, 596188160 bytes 2023-12-10T18:54:27.245 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1700 IOs, 619582464 bytes 2023-12-10T18:54:27.477 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1800 IOs, 668220416 bytes 2023-12-10T18:54:27.737 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1900 IOs, 690008064 bytes 2023-12-10T18:54:27.950 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2000 IOs, 760693760 bytes 2023-12-10T18:54:28.103 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2100 IOs, 799596544 bytes 2023-12-10T18:54:28.227 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2200 IOs, 833379328 bytes 2023-12-10T18:54:28.435 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2300 IOs, 915608576 bytes 2023-12-10T18:54:28.585 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2400 IOs, 934078464 bytes 2023-12-10T18:54:28.715 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2500 IOs, 963780608 bytes 2023-12-10T18:54:28.915 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2600 IOs, 1041762304 bytes 2023-12-10T18:54:29.068 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2700 IOs, 1063977984 bytes 2023-12-10T18:54:29.141 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2800 IOs, 1067163648 bytes 2023-12-10T18:54:29.218 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2900 IOs, 1072544768 bytes 2023-12-10T18:54:29.496 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3000 IOs, 1173716992 bytes 2023-12-10T18:54:29.583 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3100 IOs, 1179653120 bytes 2023-12-10T18:54:29.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3200 IOs, 1202959360 bytes 2023-12-10T18:54:29.791 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3300 IOs, 1212814336 bytes 2023-12-10T18:54:30.030 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3400 IOs, 1301997568 bytes 2023-12-10T18:54:30.265 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3500 IOs, 1404423168 bytes 2023-12-10T18:54:30.399 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3600 IOs, 1422866432 bytes 2023-12-10T18:54:30.513 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3700 IOs, 1461577728 bytes 2023-12-10T18:54:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3800 IOs, 1488333824 bytes 2023-12-10T18:54:30.780 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3900 IOs, 1513669632 bytes 2023-12-10T18:54:30.903 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4000 IOs, 1540114432 bytes 2023-12-10T18:54:30.967 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4100 IOs, 1542124544 bytes 2023-12-10T18:54:31.103 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4200 IOs, 1584636928 bytes 2023-12-10T18:54:31.260 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4300 IOs, 1625628672 bytes 2023-12-10T18:54:31.394 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4400 IOs, 1653617664 bytes 2023-12-10T18:54:31.490 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4500 IOs, 1672683520 bytes 2023-12-10T18:54:31.550 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4600 IOs, 1673324544 bytes 2023-12-10T18:54:31.670 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4700 IOs, 1706258432 bytes 2023-12-10T18:54:31.807 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4800 IOs, 1730870272 bytes 2023-12-10T18:54:31.896 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4900 IOs, 1753145344 bytes 2023-12-10T18:54:32.003 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5000 IOs, 1783129088 bytes 2023-12-10T18:54:32.164 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5100 IOs, 1813320704 bytes 2023-12-10T18:54:32.322 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5200 IOs, 1861147648 bytes 2023-12-10T18:54:32.402 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5300 IOs, 1875633152 bytes 2023-12-10T18:54:32.740 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5400 IOs, 1890669568 bytes 2023-12-10T18:54:32.938 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5500 IOs, 1952829440 bytes 2023-12-10T18:54:33.081 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5600 IOs, 1981189120 bytes 2023-12-10T18:54:33.158 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5700 IOs, 1990461440 bytes 2023-12-10T18:54:33.308 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5800 IOs, 2021151744 bytes 2023-12-10T18:54:33.535 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5900 IOs, 2097292288 bytes 2023-12-10T18:54:33.773 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6000 IOs, 2176744448 bytes 2023-12-10T18:54:33.953 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6100 IOs, 2227030016 bytes 2023-12-10T18:54:34.044 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6200 IOs, 2227635200 bytes 2023-12-10T18:54:34.199 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6300 IOs, 2266314752 bytes 2023-12-10T18:54:34.350 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6400 IOs, 2301090816 bytes 2023-12-10T18:54:34.419 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6500 IOs, 2301712384 bytes 2023-12-10T18:54:34.502 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6600 IOs, 2309628928 bytes 2023-12-10T18:54:34.571 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6700 IOs, 2310235136 bytes 2023-12-10T18:54:34.749 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6800 IOs, 2358633472 bytes 2023-12-10T18:54:34.897 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6900 IOs, 2390255616 bytes 2023-12-10T18:54:35.000 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:7000 IOs, 2404240384 bytes 2023-12-10T18:54:35.116 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:7100 IOs, 2435133440 bytes 2023-12-10T18:54:35.264 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:7200 IOs, 2493961216 bytes 2023-12-10T18:54:35.458 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:7300 IOs, 2560506880 bytes 2023-12-10T18:54:35.680 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=124 2023-12-10T18:54:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T18:54:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 124 -eq 124 ] 2023-12-10T18:54:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:54:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-10T18:54:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:54:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:54:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:54:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state_pattern=up+replaying 2023-12-10T18:54:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local description_pattern= 2023-12-10T18:54:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local service_pattern= 2023-12-10T18:54:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T18:54:36.682 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-10T18:54:36.682 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:54:36.683 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:54:36.683 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:54:36.683 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state_pattern=up+replaying 2023-12-10T18:54:36.683 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local description_pattern= 2023-12-10T18:54:36.683 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local service_pattern= 2023-12-10T18:54:36.683 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-test.mirror_status 2023-12-10T18:54:36.683 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-test.mirror_status 2023-12-10T18:54:36.683 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T18:54:36.684 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T18:54:36.685 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2023-12-10T18:54:36.685 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ tee /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T18:54:36.763 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:test: 2023-12-10T18:54:36.763 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: global_id: c93b253b-04a0-4ada-b252-fcc724b6756e 2023-12-10T18:54:36.763 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: state: up+replaying 2023-12-10T18:54:36.763 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":266401792.0,"last_snapshot_bytes":532803584,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1702234410,"remote_snapshot_timestamp":1702234410,"replay_state":"idle"} 2023-12-10T18:54:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: service: mirror.2 on smithi076 2023-12-10T18:54:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: last_update: 2023-12-10 18:54:14 2023-12-10T18:54:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: peer_sites: 2023-12-10T18:54:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: name: cluster2 2023-12-10T18:54:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: state: up+stopped 2023-12-10T18:54:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: description: local image is primary 2023-12-10T18:54:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: last_update: 2023-12-10 18:54:18 2023-12-10T18:54:36.772 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T18:54:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: state: up+replaying 2023-12-10T18:54:36.774 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep ^ description: .* /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T18:54:36.774 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":266401792.0,"last_snapshot_bytes":532803584,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1702234410,"remote_snapshot_timestamp":1702234410,"replay_state":"idle"} 2023-12-10T18:54:36.775 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T18:54:36.775 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo up+replaying 2023-12-10T18:54:36.775 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep ^up+ 2023-12-10T18:54:36.775 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:up+replaying 2023-12-10T18:54:36.776 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep service: *mirror..* on /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T18:54:36.776 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: service: mirror.2 on smithi076 2023-12-10T18:54:36.777 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T18:54:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local last_update=2023-12-10 18:54:14 2023-12-10T18:54:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying 2023-12-10 18:54:14 2023-12-10T18:54:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:54:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:54:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:54:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state_pattern=up+replaying 2023-12-10T18:54:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local prev_last_update=2023-12-10 18:54:14 2023-12-10T18:54:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror.mirror_status 2023-12-10T18:54:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror.mirror_status 2023-12-10T18:54:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T18:54:36.780 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror.mirror_status 2023-12-10T18:54:36.780 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-10T18:54:36.832 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: failed to query services: (13) Permission denied 2023-12-10T18:54:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local last_update state 2023-12-10T18:54:36.876 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2023-12-10T18:54:36.878 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ last_update=2023-12-10 18:54:14 2023-12-10T18:54:36.878 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2023-12-10T18:54:36.880 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ state=up+replaying 2023-12-10T18:54:36.880 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo up+replaying 2023-12-10T18:54:36.880 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep up+replaying 2023-12-10T18:54:36.881 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:up+replaying 2023-12-10T18:54:36.881 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:54:36.882 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:54:36.882 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ snap_name=snap3 2023-12-10T18:54:36.882 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror test snap3 2023-12-10T18:54:36.882 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T18:54:36.882 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:54:36.882 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:54:36.882 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap3 2023-12-10T18:54:36.882 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3 --debug-rbd=20 --debug-journaler=20 2023-12-10T18:55:05.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-10T18:55:05.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:55:05.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:55:05.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:55:05.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-10T18:55:05.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:55:05.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:55:05.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:55:05.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=started 2023-12-10T18:55:05.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T18:55:05.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T18:55:06.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-10T18:55:06.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:55:06.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:55:06.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:55:06.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=started 2023-12-10T18:55:06.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_result 2023-12-10T18:55:06.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=stopped 2023-12-10T18:55:06.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-10T18:55:06.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_instance=cluster1 2023-12-10T18:55:06.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ + shift 2023-12-10T18:55:06.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:55:06.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance=cluster1grep 2023-12-10T18:55:06.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: -i state+ 2023-12-10T18:55:06.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:local loop_instance 2023-12-10T18:55:06.676 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T18:55:06.676 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T18:55:06.677 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T18:55:06.677 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-10T18:55:06.677 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T18:55:06.677 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T18:55:06.677 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:55:06.677 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T18:55:06.677 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T18:55:06.677 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T18:55:06.678 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:55:06.678 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:55:06.678 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T18:55:06.678 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T18:55:06.678 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T18:55:06.678 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:55:06.678 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T18:55:06.678 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T18:55:06.678 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T18:55:06.678 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T18:55:06.678 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:55:06.678 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:55:06.678 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T18:55:06.679 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T18:55:06.679 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T18:55:06.679 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:55:06.679 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:55:06.679 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T18:55:06.679 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T18:55:06.679 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T18:55:06.679 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T18:55:06.679 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T18:55:06.679 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T18:55:06.679 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T18:55:06.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T18:55:06.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T18:55:06.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T18:55:06.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-10T18:55:06.800 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-10T18:55:06.800 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T18:55:06.800 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T18:55:06.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-10T18:55:06.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:1 2023-12-10T18:55:06.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T18:55:06.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:55:06.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-10T18:55:06.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T18:55:06.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T18:55:06.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:55:06.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:55:06.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T18:55:06.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T18:55:06.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T18:55:06.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:55:06.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T18:55:06.803 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T18:55:06.803 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-10T18:55:06.803 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:1 2023-12-10T18:55:06.803 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:55:06.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:55:06.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-10T18:55:06.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T18:55:06.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T18:55:06.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:55:06.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:55:06.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T18:55:06.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T18:55:06.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T18:55:06.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T18:55:06.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T18:55:06.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T18:55:06.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-10T18:55:06.827 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T18:55:06.827 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T18:55:06.827 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T18:55:06.827 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-10T18:55:06.924 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-10T18:55:06.924 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T18:55:06.924 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T18:55:06.926 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-10T18:55:06.926 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:2 2023-12-10T18:55:06.926 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T18:55:06.927 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:55:06.927 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-10T18:55:06.927 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T18:55:06.927 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T18:55:06.927 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:55:06.927 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:55:06.927 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T18:55:06.927 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T18:55:06.927 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T18:55:06.927 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:55:06.927 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T18:55:06.927 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T18:55:06.927 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-10T18:55:06.927 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:2 2023-12-10T18:55:06.927 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:55:06.928 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:55:06.928 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-10T18:55:06.928 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T18:55:06.928 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T18:55:06.928 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:55:06.928 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:55:06.928 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T18:55:06.928 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T18:55:06.928 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T18:55:06.928 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T18:55:06.928 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T18:55:06.928 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T18:55:06.928 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-10T18:55:06.946 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T18:55:06.946 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T18:55:06.946 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T18:55:06.946 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-10T18:55:07.074 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:55:07.074 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ status_result= "state": "Replaying" 2023-12-10T18:55:07.074 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo "state": "Replaying" 2023-12-10T18:55:07.075 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep -i Replaying 2023-12-10T18:55:07.075 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "state": "Replaying" 2023-12-10T18:55:07.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=started 2023-12-10T18:55:07.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test started = started 2023-12-10T18:55:07.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:55:07.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-10T18:55:07.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T18:55:07.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T18:55:07.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:55:07.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:55:07.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = journal ] 2023-12-10T18:55:07.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = snapshot ] 2023-12-10T18:55:07.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-10T18:55:07.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T18:55:07.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T18:55:07.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:55:07.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:55:07.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster2-mirror-test.status 2023-12-10T18:55:07.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2-mirror-test.status 2023-12-10T18:55:07.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T18:55:07.078 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-test.status 2023-12-10T18:55:07.078 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-test.status 2023-12-10T18:55:07.078 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-test.status 2023-12-10T18:55:07.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T18:55:07.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:55:07.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-10T18:55:07.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T18:55:07.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:55:07.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:55:07.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-10T18:55:07.605 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.4429065886397411) > weight (0.25) 2023-12-10T18:55:07.605 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.9109053134980017) > weight (0.25) 2023-12-10T18:55:07.605 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.7663063746055696) > weight (0.25) 2023-12-10T18:55:07.605 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.3378508152008104) > weight (0.25) 2023-12-10T18:55:07.605 INFO:tasks.rbd_mirror_thrash:waiting for 102 secs before thrashing 2023-12-10T18:55:08.681 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Snapshot ID: 13 2023-12-10T18:55:08.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.status 2023-12-10T18:55:08.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T18:55:08.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:55:08.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:55:08.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-test.status 2023-12-10T18:55:08.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-10T18:55:08.704 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T18:55:08.775 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-10T18:55:08.778 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snapshot_id=13 2023-12-10T18:55:08.778 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T18:55:08.778 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.2 2023-12-10T18:55:08.979 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:55:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:55:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:55:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:55:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:55:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-10T18:55:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T18:55:09.054 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T18:55:09.054 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T18:55:09.057 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T18:55:09.057 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T18:55:09.057 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T18:55:09.057 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T18:55:09.057 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 13 = 2023-12-10T18:55:09.057 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.4 2023-12-10T18:55:09.458 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:55:09.458 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:55:09.458 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:55:09.458 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:55:09.459 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:55:09.459 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-10T18:55:09.459 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T18:55:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T18:55:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T18:55:09.535 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T18:55:09.535 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T18:55:09.535 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T18:55:09.535 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T18:55:09.535 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 13 = 2023-12-10T18:55:09.536 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.8 2023-12-10T18:55:10.337 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:55:10.337 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:55:10.337 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:55:10.337 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:55:10.337 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:55:10.337 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-10T18:55:10.337 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T18:55:10.497 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T18:55:10.497 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T18:55:10.501 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T18:55:10.501 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T18:55:10.501 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T18:55:10.502 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T18:55:10.502 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 13 = 2023-12-10T18:55:10.502 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1.6 2023-12-10T18:55:12.103 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:55:12.104 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:55:12.104 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:55:12.104 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:55:12.104 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:55:12.104 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-10T18:55:12.104 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T18:55:12.170 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T18:55:12.173 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=13 2023-12-10T18:55:12.173 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 13 = 13 2023-12-10T18:55:12.174 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:55:12.174 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snap_present cluster1 mirror test snap3 2023-12-10T18:55:12.174 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:55:12.174 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:55:12.174 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:55:12.174 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap3 2023-12-10T18:55:12.174 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T18:55:12.174 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T18:55:13.174 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 info mirror/test@snap3 2023-12-10T18:55:13.240 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'test': 2023-12-10T18:55:13.241 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 512 MiB in 128 objects 2023-12-10T18:55:13.241 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T18:55:13.241 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 6 2023-12-10T18:55:13.241 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 10c0be4f5681 2023-12-10T18:55:13.241 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.10c0be4f5681 2023-12-10T18:55:13.241 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T18:55:13.241 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering, non-primary 2023-12-10T18:55:13.241 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T18:55:13.241 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T18:55:13.241 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 18:52:46 2023 2023-12-10T18:55:13.241 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 18:54:19 2023 2023-12-10T18:55:13.241 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 18:52:46 2023 2023-12-10T18:55:13.241 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: protected: False 2023-12-10T18:55:13.241 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T18:55:13.242 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T18:55:13.242 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: c93b253b-04a0-4ada-b252-fcc724b6756e 2023-12-10T18:55:13.242 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: false 2023-12-10T18:55:13.245 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:55:13.245 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n snap2-clean ] 2023-12-10T18:55:13.245 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror test snap2-clean 2023-12-10T18:55:13.245 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:55:13.245 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:55:13.245 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap2-clean 2023-12-10T18:55:13.245 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T18:55:13.246 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T18:55:13.246 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:55:13.246 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:55:13.246 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T18:55:13.933 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T18:55:13.945 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:55:15.078 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T18:55:15.095 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:55:15.375 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:55:15.511 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:55:15.511 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror test snap3 2023-12-10T18:55:15.511 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:55:15.511 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:55:15.511 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap3 2023-12-10T18:55:15.511 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T18:55:15.511 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T18:55:15.511 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:55:15.511 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:55:15.512 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T18:55:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T18:55:16.194 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:55:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T18:55:17.314 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:55:17.594 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:55:17.700 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.8028687954498469) > weight (0.25) 2023-12-10T18:55:17.701 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.7099487250644303) > weight (0.25) 2023-12-10T18:55:17.701 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.7077980886914323) > weight (0.25) 2023-12-10T18:55:17.701 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2023-12-10T18:55:17.701 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2023-12-10T18:55:17.701 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2023-12-10T18:55:17.702 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi076.stderr:2023-12-10T18:55:17.698+0000 7fd2e2f96640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 39939) UID: 1000 2023-12-10T18:55:17.742 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:55:17.743 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ clean_snap_name=snap3-clean 2023-12-10T18:55:17.743 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror test snap3-clean 2023-12-10T18:55:17.743 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T18:55:17.743 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:55:17.743 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:55:17.743 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap3-clean 2023-12-10T18:55:17.743 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3-clean --debug-rbd=20 --debug-journaler=20 2023-12-10T18:55:18.647 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stress_write_image cluster2 mirror test 2023-12-10T18:55:18.647 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T18:55:18.647 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:55:18.647 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:55:18.647 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-10T18:55:18.649 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local duration=9 2023-12-10T18:55:18.649 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T18:55:18.649 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-10T18:55:18.903 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:100 IOs, 16992256 bytes 2023-12-10T18:55:19.129 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:200 IOs, 68337664 bytes 2023-12-10T18:55:19.370 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:300 IOs, 145764352 bytes 2023-12-10T18:55:19.554 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:400 IOs, 196253696 bytes 2023-12-10T18:55:19.642 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:500 IOs, 218400768 bytes 2023-12-10T18:55:19.753 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:600 IOs, 222813184 bytes 2023-12-10T18:55:19.850 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:700 IOs, 234450944 bytes 2023-12-10T18:55:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:800 IOs, 351318016 bytes 2023-12-10T18:55:20.261 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:900 IOs, 360213504 bytes 2023-12-10T18:55:20.354 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1000 IOs, 372431872 bytes 2023-12-10T18:55:20.561 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1100 IOs, 452978688 bytes 2023-12-10T18:55:20.702 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2023-12-10T18:55:20.702 INFO:teuthology.orchestra.run:waiting for 600 2023-12-10T18:55:20.703 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2023-12-10T18:55:20.703 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2023-12-10T18:55:20.703 DEBUG:teuthology.orchestra.run.smithi076:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2023-12-10T18:55:20.705 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2023-12-10T18:55:20.705 INFO:tasks.rbd_mirror_thrash:waiting for 107 secs before thrashing 2023-12-10T18:55:20.721 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1200 IOs, 489155584 bytes 2023-12-10T18:55:20.860 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1300 IOs, 515720192 bytes 2023-12-10T18:55:20.956 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1400 IOs, 518172672 bytes 2023-12-10T18:55:21.081 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1500 IOs, 539546624 bytes 2023-12-10T18:55:21.317 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1600 IOs, 617523200 bytes 2023-12-10T18:55:21.424 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1700 IOs, 636157952 bytes 2023-12-10T18:55:21.555 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1800 IOs, 666108928 bytes 2023-12-10T18:55:21.747 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1900 IOs, 727950336 bytes 2023-12-10T18:55:21.978 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2000 IOs, 806936576 bytes 2023-12-10T18:55:22.083 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2100 IOs, 825454592 bytes 2023-12-10T18:55:22.218 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2200 IOs, 860447744 bytes 2023-12-10T18:55:22.349 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2300 IOs, 890085376 bytes 2023-12-10T18:55:22.415 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2400 IOs, 912083968 bytes 2023-12-10T18:55:22.608 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2500 IOs, 968550400 bytes 2023-12-10T18:55:22.870 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2600 IOs, 1059173376 bytes 2023-12-10T18:55:23.069 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2700 IOs, 1110828032 bytes 2023-12-10T18:55:23.165 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2800 IOs, 1125288960 bytes 2023-12-10T18:55:23.415 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2900 IOs, 1216687104 bytes 2023-12-10T18:55:23.558 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3000 IOs, 1254659072 bytes 2023-12-10T18:55:23.671 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3100 IOs, 1275924480 bytes 2023-12-10T18:55:23.917 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3200 IOs, 1356485632 bytes 2023-12-10T18:55:24.031 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3300 IOs, 1371381760 bytes 2023-12-10T18:55:24.177 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3400 IOs, 1400975360 bytes 2023-12-10T18:55:24.258 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3500 IOs, 1416251392 bytes 2023-12-10T18:55:24.384 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3600 IOs, 1452556288 bytes 2023-12-10T18:55:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3700 IOs, 1502425088 bytes 2023-12-10T18:55:24.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3800 IOs, 1522015232 bytes 2023-12-10T18:55:24.783 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3900 IOs, 1522626560 bytes 2023-12-10T18:55:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4000 IOs, 1584443392 bytes 2023-12-10T18:55:25.254 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4100 IOs, 1666802688 bytes 2023-12-10T18:55:25.363 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4200 IOs, 1706924032 bytes 2023-12-10T18:55:25.562 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4300 IOs, 1751226368 bytes 2023-12-10T18:55:25.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4400 IOs, 1770061824 bytes 2023-12-10T18:55:25.873 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4500 IOs, 1832853504 bytes 2023-12-10T18:55:25.987 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4600 IOs, 1852045312 bytes 2023-12-10T18:55:26.060 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4700 IOs, 1858036736 bytes 2023-12-10T18:55:26.304 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4800 IOs, 1946010624 bytes 2023-12-10T18:55:26.517 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4900 IOs, 2013628416 bytes 2023-12-10T18:55:26.672 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5000 IOs, 2059143168 bytes 2023-12-10T18:55:26.782 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5100 IOs, 2075011072 bytes 2023-12-10T18:55:26.995 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5200 IOs, 2161019904 bytes 2023-12-10T18:55:27.190 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5300 IOs, 2204750848 bytes 2023-12-10T18:55:27.313 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5400 IOs, 2225844224 bytes 2023-12-10T18:55:27.477 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5500 IOs, 2272807936 bytes 2023-12-10T18:55:27.654 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=124 2023-12-10T18:55:27.655 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T18:55:27.655 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 124 -eq 124 ] 2023-12-10T18:55:27.655 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:55:27.655 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-10T18:55:27.655 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:55:27.655 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:55:27.655 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:55:27.655 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state_pattern=up+replaying 2023-12-10T18:55:27.655 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local description_pattern= 2023-12-10T18:55:27.655 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local service_pattern= 2023-12-10T18:55:27.655 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T18:55:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-10T18:55:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:55:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:55:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:55:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state_pattern=up+replaying 2023-12-10T18:55:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local description_pattern= 2023-12-10T18:55:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local service_pattern= 2023-12-10T18:55:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-test.mirror_status 2023-12-10T18:55:28.657 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-test.mirror_status 2023-12-10T18:55:28.657 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T18:55:28.658 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T18:55:28.658 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2023-12-10T18:55:28.658 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ tee /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T18:55:28.743 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:test: 2023-12-10T18:55:28.743 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: global_id: c93b253b-04a0-4ada-b252-fcc724b6756e 2023-12-10T18:55:28.743 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: state: up+replaying 2023-12-10T18:55:28.743 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: description: replaying, {"bytes_per_second":17613414.4,"bytes_per_snapshot":500645376.0,"last_snapshot_bytes":528402432,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1702234507,"remote_snapshot_timestamp":1702234507,"replay_state":"idle"} 2023-12-10T18:55:28.743 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: service: mirror.2 on smithi076 2023-12-10T18:55:28.743 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: last_update: 2023-12-10 18:55:14 2023-12-10T18:55:28.743 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: peer_sites: 2023-12-10T18:55:28.743 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: name: cluster2 2023-12-10T18:55:28.743 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: state: up+stopped 2023-12-10T18:55:28.743 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: description: local image is primary 2023-12-10T18:55:28.743 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: last_update: 2023-12-10 18:55:18 2023-12-10T18:55:28.751 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T18:55:28.752 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: state: up+replaying 2023-12-10T18:55:28.753 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep ^ description: .* /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T18:55:28.753 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: description: replaying, {"bytes_per_second":17613414.4,"bytes_per_snapshot":500645376.0,"last_snapshot_bytes":528402432,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1702234507,"remote_snapshot_timestamp":1702234507,"replay_state":"idle"} 2023-12-10T18:55:28.754 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T18:55:28.754 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo up+replaying 2023-12-10T18:55:28.754 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep ^up+ 2023-12-10T18:55:28.755 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:up+replaying 2023-12-10T18:55:28.755 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep service: *mirror..* on /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T18:55:28.756 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: service: mirror.2 on smithi076 2023-12-10T18:55:28.756 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T18:55:28.758 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local last_update=2023-12-10 18:55:14 2023-12-10T18:55:28.758 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying 2023-12-10 18:55:14 2023-12-10T18:55:28.759 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:55:28.759 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:55:28.759 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:55:28.759 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state_pattern=up+replaying 2023-12-10T18:55:28.759 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local prev_last_update=2023-12-10 18:55:14 2023-12-10T18:55:28.759 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror.mirror_status 2023-12-10T18:55:28.759 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror.mirror_status 2023-12-10T18:55:28.759 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T18:55:28.760 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror.mirror_status 2023-12-10T18:55:28.760 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-10T18:55:28.814 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: failed to query services: (13) Permission denied 2023-12-10T18:55:28.856 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local last_update state 2023-12-10T18:55:28.856 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2023-12-10T18:55:28.860 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ last_update=2023-12-10 18:55:14 2023-12-10T18:55:28.860 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2023-12-10T18:55:28.863 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ state=up+replaying 2023-12-10T18:55:28.863 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep up+replaying 2023-12-10T18:55:28.863 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo up+replaying 2023-12-10T18:55:28.863 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:up+replaying 2023-12-10T18:55:28.864 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:55:28.864 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:55:28.864 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ snap_name=snap4 2023-12-10T18:55:28.864 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror test snap4 2023-12-10T18:55:28.864 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T18:55:28.864 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:55:28.864 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:55:28.864 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap4 2023-12-10T18:55:28.864 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4 --debug-rbd=20 --debug-journaler=20 2023-12-10T18:55:56.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-10T18:55:56.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:55:56.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:55:56.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:55:56.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-10T18:55:56.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:55:56.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:55:56.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:55:56.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=started 2023-12-10T18:55:56.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T18:55:56.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T18:55:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-10T18:55:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:55:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:55:57.699 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:55:57.699 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=started 2023-12-10T18:55:57.699 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_result 2023-12-10T18:55:57.699 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=stopped 2023-12-10T18:55:57.699 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemons cluster1 rbd+ mirror status mirror/test 2023-12-10T18:55:57.699 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:grep -i state 2023-12-10T18:55:57.700 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_instance=cluster1 2023-12-10T18:55:57.700 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T18:55:57.700 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:55:57.700 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance=cluster1 2023-12-10T18:55:57.700 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loop_instance 2023-12-10T18:55:57.700 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T18:55:57.700 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T18:55:57.700 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T18:55:57.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-10T18:55:57.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T18:55:57.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T18:55:57.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:55:57.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T18:55:57.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T18:55:57.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T18:55:57.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:55:57.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:55:57.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T18:55:57.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T18:55:57.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T18:55:57.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:55:57.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T18:55:57.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T18:55:57.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T18:55:57.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T18:55:57.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:55:57.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:55:57.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T18:55:57.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T18:55:57.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T18:55:57.704 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:55:57.704 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:55:57.704 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T18:55:57.704 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T18:55:57.704 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T18:55:57.704 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T18:55:57.704 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T18:55:57.704 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T18:55:57.704 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T18:55:57.724 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T18:55:57.724 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T18:55:57.724 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T18:55:57.724 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-10T18:55:57.834 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-10T18:55:57.834 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T18:55:57.834 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T18:55:57.836 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-10T18:55:57.836 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:1 2023-12-10T18:55:57.836 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T18:55:57.837 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:55:57.837 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-10T18:55:57.837 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T18:55:57.837 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T18:55:57.837 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:55:57.837 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:55:57.837 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T18:55:57.837 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T18:55:57.837 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T18:55:57.837 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:55:57.837 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T18:55:57.837 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T18:55:57.837 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-10T18:55:57.837 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:1 2023-12-10T18:55:57.837 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:55:57.837 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:55:57.838 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-10T18:55:57.838 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T18:55:57.838 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T18:55:57.838 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:55:57.838 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:55:57.838 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T18:55:57.839 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T18:55:57.839 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T18:55:57.839 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T18:55:57.839 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T18:55:57.839 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T18:55:57.839 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-10T18:55:57.862 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T18:55:57.862 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T18:55:57.862 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T18:55:57.862 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-10T18:55:57.975 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-10T18:55:57.975 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T18:55:57.975 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T18:55:57.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-10T18:55:57.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:2 2023-12-10T18:55:57.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T18:55:57.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:55:57.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-10T18:55:57.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T18:55:57.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T18:55:57.978 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:55:57.978 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:55:57.978 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T18:55:57.978 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T18:55:57.978 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T18:55:57.978 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:55:57.978 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T18:55:57.978 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T18:55:57.978 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-10T18:55:57.978 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:2 2023-12-10T18:55:57.978 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:55:57.978 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:55:57.978 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-10T18:55:57.978 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T18:55:57.978 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T18:55:57.979 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:55:57.979 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:55:57.979 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T18:55:57.979 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T18:55:57.979 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T18:55:57.979 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T18:55:57.979 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T18:55:57.979 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T18:55:57.979 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-10T18:55:58.001 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T18:55:58.001 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T18:55:58.001 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T18:55:58.001 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-10T18:55:58.125 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:55:58.125 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ status_result= "state": "Replaying" 2023-12-10T18:55:58.126 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo "state": "Replaying" 2023-12-10T18:55:58.126 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep -i Replaying 2023-12-10T18:55:58.127 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "state": "Replaying" 2023-12-10T18:55:58.130 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=started 2023-12-10T18:55:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test started = started 2023-12-10T18:55:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:55:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-10T18:55:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T18:55:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T18:55:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:55:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:55:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = journal ] 2023-12-10T18:55:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = snapshot ] 2023-12-10T18:55:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-10T18:55:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T18:55:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T18:55:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:55:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:55:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster2-mirror-test.status 2023-12-10T18:55:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2-mirror-test.status 2023-12-10T18:55:58.132 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T18:55:58.132 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-test.status 2023-12-10T18:55:58.132 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-test.status 2023-12-10T18:55:58.132 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-test.status 2023-12-10T18:55:58.132 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T18:55:58.132 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:55:58.132 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-10T18:55:58.132 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T18:55:58.132 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:55:58.133 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:55:58.133 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-10T18:55:59.705 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Snapshot ID: 17 2023-12-10T18:55:59.720 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.status 2023-12-10T18:55:59.720 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T18:55:59.720 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:55:59.720 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:55:59.720 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-test.status 2023-12-10T18:55:59.720 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T18:55:59.721 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-10T18:55:59.796 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-10T18:55:59.799 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snapshot_id=17 2023-12-10T18:55:59.800 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T18:55:59.800 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.2 2023-12-10T18:56:00.000 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:56:00.000 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:56:00.001 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:56:00.001 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:56:00.001 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:56:00.001 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-10T18:56:00.001 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T18:56:00.071 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T18:56:00.071 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T18:56:00.074 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T18:56:00.074 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T18:56:00.074 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T18:56:00.074 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T18:56:00.074 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 17 = 2023-12-10T18:56:00.074 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.4 2023-12-10T18:56:00.476 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:56:00.476 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:56:00.476 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:56:00.477 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:56:00.477 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:56:00.477 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-10T18:56:00.477 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T18:56:00.556 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T18:56:00.556 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T18:56:00.558 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T18:56:00.558 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T18:56:00.558 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T18:56:00.559 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T18:56:00.559 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 17 = 2023-12-10T18:56:00.559 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.8 2023-12-10T18:56:01.360 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:56:01.360 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:56:01.360 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:56:01.360 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:56:01.360 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:56:01.360 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-10T18:56:01.360 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T18:56:01.445 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T18:56:01.446 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T18:56:01.448 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T18:56:01.448 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T18:56:01.448 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T18:56:01.448 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T18:56:01.448 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 17 = 2023-12-10T18:56:01.448 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1.6 2023-12-10T18:56:03.049 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:56:03.050 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:56:03.050 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:56:03.050 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:56:03.050 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:56:03.050 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-10T18:56:03.050 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T18:56:03.176 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T18:56:03.180 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=17 2023-12-10T18:56:03.180 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 17 = 17 2023-12-10T18:56:03.180 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:56:03.180 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snap_present cluster1 mirror test snap4 2023-12-10T18:56:03.181 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:56:03.181 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:56:03.181 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:56:03.181 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap4 2023-12-10T18:56:03.181 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T18:56:03.181 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T18:56:04.181 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 info mirror/test@snap4 2023-12-10T18:56:04.245 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'test': 2023-12-10T18:56:04.245 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 512 MiB in 128 objects 2023-12-10T18:56:04.245 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T18:56:04.245 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 8 2023-12-10T18:56:04.245 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 10c0be4f5681 2023-12-10T18:56:04.246 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.10c0be4f5681 2023-12-10T18:56:04.246 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T18:56:04.246 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering, non-primary 2023-12-10T18:56:04.246 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T18:56:04.246 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T18:56:04.246 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 18:52:46 2023 2023-12-10T18:56:04.246 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 18:54:19 2023 2023-12-10T18:56:04.246 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 18:52:46 2023 2023-12-10T18:56:04.246 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: protected: False 2023-12-10T18:56:04.246 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T18:56:04.246 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T18:56:04.246 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: c93b253b-04a0-4ada-b252-fcc724b6756e 2023-12-10T18:56:04.246 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: false 2023-12-10T18:56:04.250 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:56:04.250 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n snap3-clean ] 2023-12-10T18:56:04.250 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror test snap3-clean 2023-12-10T18:56:04.250 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:56:04.250 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:56:04.250 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap3-clean 2023-12-10T18:56:04.250 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T18:56:04.250 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T18:56:04.250 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:56:04.250 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:56:04.250 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T18:56:04.965 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T18:56:04.975 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:56:06.184 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T18:56:06.205 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:56:06.486 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:56:06.621 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:56:06.621 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror test snap4 2023-12-10T18:56:06.621 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:56:06.621 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:56:06.621 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap4 2023-12-10T18:56:06.621 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T18:56:06.621 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T18:56:06.621 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:56:06.622 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:56:06.622 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T18:56:07.307 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T18:56:07.327 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:56:08.529 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T18:56:08.551 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:56:08.836 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:56:08.974 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:56:08.974 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ clean_snap_name=snap4-clean 2023-12-10T18:56:08.974 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror test snap4-clean 2023-12-10T18:56:08.974 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T18:56:08.974 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:56:08.974 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:56:08.974 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap4-clean 2023-12-10T18:56:08.974 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4-clean --debug-rbd=20 --debug-journaler=20 2023-12-10T18:56:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stress_write_image cluster2 mirror test 2023-12-10T18:56:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T18:56:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:56:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:56:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-10T18:56:09.666 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local duration=14 2023-12-10T18:56:09.666 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T18:56:09.666 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-10T18:56:09.969 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:100 IOs, 42701824 bytes 2023-12-10T18:56:10.214 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:200 IOs, 117630976 bytes 2023-12-10T18:56:10.454 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:300 IOs, 200484864 bytes 2023-12-10T18:56:10.716 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:400 IOs, 298582016 bytes 2023-12-10T18:56:10.819 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:500 IOs, 304288768 bytes 2023-12-10T18:56:11.016 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:600 IOs, 379205632 bytes 2023-12-10T18:56:11.169 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:700 IOs, 410419200 bytes 2023-12-10T18:56:11.312 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:800 IOs, 445356032 bytes 2023-12-10T18:56:11.426 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:900 IOs, 473764864 bytes 2023-12-10T18:56:11.555 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1000 IOs, 503484416 bytes 2023-12-10T18:56:11.807 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1100 IOs, 586458112 bytes 2023-12-10T18:56:11.950 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1200 IOs, 640548864 bytes 2023-12-10T18:56:12.081 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1300 IOs, 651851776 bytes 2023-12-10T18:56:12.175 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1400 IOs, 665893888 bytes 2023-12-10T18:56:12.314 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1500 IOs, 701283328 bytes 2023-12-10T18:56:12.382 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1600 IOs, 701901824 bytes 2023-12-10T18:56:12.480 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1700 IOs, 712602624 bytes 2023-12-10T18:56:12.658 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1800 IOs, 763307008 bytes 2023-12-10T18:56:12.902 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1900 IOs, 853991424 bytes 2023-12-10T18:56:13.157 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2000 IOs, 915909632 bytes 2023-12-10T18:56:13.241 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2100 IOs, 928037888 bytes 2023-12-10T18:56:13.394 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2200 IOs, 966409216 bytes 2023-12-10T18:56:13.476 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2300 IOs, 972694528 bytes 2023-12-10T18:56:13.574 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2400 IOs, 986134528 bytes 2023-12-10T18:56:13.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2500 IOs, 1022046208 bytes 2023-12-10T18:56:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2600 IOs, 1079340032 bytes 2023-12-10T18:56:14.027 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2700 IOs, 1100179456 bytes 2023-12-10T18:56:14.156 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2800 IOs, 1133772800 bytes 2023-12-10T18:56:14.329 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2900 IOs, 1195260928 bytes 2023-12-10T18:56:14.550 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3000 IOs, 1250712576 bytes 2023-12-10T18:56:14.676 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3100 IOs, 1272186880 bytes 2023-12-10T18:56:14.814 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3200 IOs, 1300423680 bytes 2023-12-10T18:56:14.936 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3300 IOs, 1324657664 bytes 2023-12-10T18:56:15.172 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3400 IOs, 1395116032 bytes 2023-12-10T18:56:15.349 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3500 IOs, 1437816832 bytes 2023-12-10T18:56:15.459 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3600 IOs, 1455221760 bytes 2023-12-10T18:56:15.561 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3700 IOs, 1500251136 bytes 2023-12-10T18:56:15.749 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3800 IOs, 1526085632 bytes 2023-12-10T18:56:15.873 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3900 IOs, 1550435328 bytes 2023-12-10T18:56:16.000 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4000 IOs, 1577368576 bytes 2023-12-10T18:56:16.193 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4100 IOs, 1643307008 bytes 2023-12-10T18:56:16.310 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4200 IOs, 1666676736 bytes 2023-12-10T18:56:16.492 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4300 IOs, 1721779200 bytes 2023-12-10T18:56:16.708 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4400 IOs, 1795475456 bytes 2023-12-10T18:56:16.844 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4500 IOs, 1841007616 bytes 2023-12-10T18:56:17.063 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4600 IOs, 1887730688 bytes 2023-12-10T18:56:17.188 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4700 IOs, 1912114176 bytes 2023-12-10T18:56:17.418 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4800 IOs, 1992824832 bytes 2023-12-10T18:56:17.604 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4900 IOs, 2025352192 bytes 2023-12-10T18:56:17.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5000 IOs, 2094750720 bytes 2023-12-10T18:56:17.905 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5100 IOs, 2099931136 bytes 2023-12-10T18:56:18.064 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5200 IOs, 2142202880 bytes 2023-12-10T18:56:18.169 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5300 IOs, 2157026304 bytes 2023-12-10T18:56:18.300 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5400 IOs, 2189426688 bytes 2023-12-10T18:56:18.521 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5500 IOs, 2258472960 bytes 2023-12-10T18:56:18.857 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5600 IOs, 2279259136 bytes 2023-12-10T18:56:19.001 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5700 IOs, 2299099136 bytes 2023-12-10T18:56:19.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5800 IOs, 2300244992 bytes 2023-12-10T18:56:19.269 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5900 IOs, 2354888704 bytes 2023-12-10T18:56:19.352 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6000 IOs, 2356140032 bytes 2023-12-10T18:56:19.478 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6100 IOs, 2391059456 bytes 2023-12-10T18:56:19.811 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6200 IOs, 2427640832 bytes 2023-12-10T18:56:19.979 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6300 IOs, 2468289536 bytes 2023-12-10T18:56:20.128 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6400 IOs, 2498864128 bytes 2023-12-10T18:56:20.209 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6500 IOs, 2499495936 bytes 2023-12-10T18:56:20.353 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6600 IOs, 2543132672 bytes 2023-12-10T18:56:20.490 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6700 IOs, 2559063040 bytes 2023-12-10T18:56:20.725 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6800 IOs, 2637451264 bytes 2023-12-10T18:56:20.840 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6900 IOs, 2659485696 bytes 2023-12-10T18:56:21.100 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:7000 IOs, 2752361472 bytes 2023-12-10T18:56:21.224 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:7100 IOs, 2778691584 bytes 2023-12-10T18:56:21.367 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:7200 IOs, 2817026048 bytes 2023-12-10T18:56:21.518 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:7300 IOs, 2849756160 bytes 2023-12-10T18:56:21.733 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:7400 IOs, 2919703552 bytes 2023-12-10T18:56:21.905 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:7500 IOs, 2976966656 bytes 2023-12-10T18:56:22.166 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:7600 IOs, 3049673728 bytes 2023-12-10T18:56:22.288 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:7700 IOs, 3074726912 bytes 2023-12-10T18:56:22.389 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:7800 IOs, 3090299904 bytes 2023-12-10T18:56:22.555 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:7900 IOs, 3134448640 bytes 2023-12-10T18:56:22.739 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:8000 IOs, 3188174848 bytes 2023-12-10T18:56:22.861 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:8100 IOs, 3206999040 bytes 2023-12-10T18:56:23.071 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:8200 IOs, 3275688960 bytes 2023-12-10T18:56:23.189 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:8300 IOs, 3298601984 bytes 2023-12-10T18:56:23.322 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:8400 IOs, 3331267584 bytes 2023-12-10T18:56:23.524 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:8500 IOs, 3382183936 bytes 2023-12-10T18:56:23.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=124 2023-12-10T18:56:23.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T18:56:23.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 124 -eq 124 ] 2023-12-10T18:56:23.676 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:56:23.676 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-10T18:56:23.676 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:56:23.676 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:56:23.676 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:56:23.676 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state_pattern=up+replaying 2023-12-10T18:56:23.676 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local description_pattern= 2023-12-10T18:56:23.676 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local service_pattern= 2023-12-10T18:56:23.676 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T18:56:24.677 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-10T18:56:24.677 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:56:24.677 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:56:24.677 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:56:24.677 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state_pattern=up+replaying 2023-12-10T18:56:24.677 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local description_pattern= 2023-12-10T18:56:24.677 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local service_pattern= 2023-12-10T18:56:24.677 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-test.mirror_status 2023-12-10T18:56:24.677 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-test.mirror_status 2023-12-10T18:56:24.677 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T18:56:24.678 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T18:56:24.679 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2023-12-10T18:56:24.679 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ tee /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T18:56:24.755 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:test: 2023-12-10T18:56:24.755 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: global_id: c93b253b-04a0-4ada-b252-fcc724b6756e 2023-12-10T18:56:24.755 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: state: up+replaying 2023-12-10T18:56:24.755 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: description: replaying, {"bytes_per_second":17509785.6,"bytes_per_snapshot":526848000.0,"last_snapshot_bytes":525293568,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1702234558,"remote_snapshot_timestamp":1702234558,"replay_state":"idle"} 2023-12-10T18:56:24.756 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: service: mirror.2 on smithi076 2023-12-10T18:56:24.756 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: last_update: 2023-12-10 18:56:14 2023-12-10T18:56:24.756 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: peer_sites: 2023-12-10T18:56:24.756 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: name: cluster2 2023-12-10T18:56:24.756 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: state: up+stopped 2023-12-10T18:56:24.756 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: description: local image is primary 2023-12-10T18:56:24.756 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: last_update: 2023-12-10 18:56:18 2023-12-10T18:56:24.799 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T18:56:24.799 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: state: up+replaying 2023-12-10T18:56:24.800 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep ^ description: .* /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T18:56:24.800 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: description: replaying, {"bytes_per_second":17509785.6,"bytes_per_snapshot":526848000.0,"last_snapshot_bytes":525293568,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1702234558,"remote_snapshot_timestamp":1702234558,"replay_state":"idle"} 2023-12-10T18:56:24.801 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T18:56:24.801 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo up+replaying 2023-12-10T18:56:24.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep ^up+ 2023-12-10T18:56:24.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:up+replaying 2023-12-10T18:56:24.803 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep service: *mirror..* on /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T18:56:24.803 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: service: mirror.2 on smithi076 2023-12-10T18:56:24.803 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T18:56:24.806 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local last_update=2023-12-10 18:56:14 2023-12-10T18:56:24.806 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying 2023-12-10 18:56:14 2023-12-10T18:56:24.806 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:56:24.806 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:56:24.806 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:56:24.806 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state_pattern=up+replaying 2023-12-10T18:56:24.807 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local prev_last_update=2023-12-10 18:56:14 2023-12-10T18:56:24.807 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror.mirror_status 2023-12-10T18:56:24.807 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror.mirror_status 2023-12-10T18:56:24.807 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T18:56:24.807 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror.mirror_status 2023-12-10T18:56:24.808 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-10T18:56:24.856 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: failed to query services: (13) Permission denied 2023-12-10T18:56:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local last_update state 2023-12-10T18:56:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2023-12-10T18:56:24.988 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ last_update=2023-12-10 18:56:14 2023-12-10T18:56:24.989 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2023-12-10T18:56:24.992 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ state=up+replaying 2023-12-10T18:56:24.992 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo up+replaying 2023-12-10T18:56:24.992 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep up+replaying 2023-12-10T18:56:24.993 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:up+replaying 2023-12-10T18:56:24.994 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:56:24.994 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:56:24.994 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ snap_name=snap5 2023-12-10T18:56:24.994 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror test snap5 2023-12-10T18:56:24.994 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T18:56:24.994 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:56:24.994 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:56:24.995 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap5 2023-12-10T18:56:24.995 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5 --debug-rbd=20 --debug-journaler=20 2023-12-10T18:56:49.606 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6669477918461031) > weight (0.25) 2023-12-10T18:56:49.607 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2023-12-10T18:56:49.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2023-12-10T18:56:49.607 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2023-12-10T18:56:49.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi076.stderr:2023-12-10T18:56:49.603+0000 7f57b94d6640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 39901) UID: 1000 2023-12-10T18:56:50.608 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2023-12-10T18:56:50.609 INFO:teuthology.orchestra.run:waiting for 600 2023-12-10T18:56:50.609 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2023-12-10T18:56:50.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-12-10T18:56:50.610 DEBUG:teuthology.orchestra.run.smithi076:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-12-10T18:56:50.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-12-10T18:56:50.613 INFO:tasks.rbd_mirror_thrash:waiting for 60 secs before thrashing 2023-12-10T18:56:53.692 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-10T18:56:53.692 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:56:53.692 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:56:53.692 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:56:53.692 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-10T18:56:53.692 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:56:53.692 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:56:53.692 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:56:53.692 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=started 2023-12-10T18:56:53.692 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T18:56:53.693 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T18:56:54.693 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-10T18:56:54.693 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:56:54.693 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:56:54.693 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:56:54.693 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=started 2023-12-10T18:56:54.693 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_result 2023-12-10T18:56:54.693 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=stopped 2023-12-10T18:56:54.694 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-10T18:56:54.694 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_instance=cluster1 2023-12-10T18:56:54.694 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T18:56:54.694 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:56:54.694 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ + local instance=cluster1 2023-12-10T18:56:54.694 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loop_instance 2023-12-10T18:56:54.695 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T18:56:54.695 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:grep -i state 2023-12-10T18:56:54.695 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T18:56:54.695 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T18:56:54.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-10T18:56:54.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T18:56:54.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T18:56:54.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:56:54.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T18:56:54.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T18:56:54.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T18:56:54.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:56:54.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:56:54.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T18:56:54.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T18:56:54.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T18:56:54.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:56:54.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T18:56:54.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T18:56:54.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T18:56:54.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T18:56:54.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:56:54.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:56:54.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T18:56:54.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T18:56:54.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T18:56:54.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:56:54.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:56:54.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T18:56:54.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T18:56:54.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T18:56:54.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T18:56:54.699 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T18:56:54.699 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T18:56:54.699 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T18:56:54.722 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T18:56:54.722 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T18:56:54.722 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T18:56:54.722 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-10T18:56:54.830 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-10T18:56:54.830 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T18:56:54.830 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T18:56:54.832 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-10T18:56:54.832 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:1 2023-12-10T18:56:54.832 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T18:56:54.832 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:56:54.832 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-10T18:56:54.832 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T18:56:54.832 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T18:56:54.832 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:56:54.832 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:56:54.832 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T18:56:54.832 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T18:56:54.832 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T18:56:54.833 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:56:54.833 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T18:56:54.833 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T18:56:54.834 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-10T18:56:54.834 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:1 2023-12-10T18:56:54.834 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:56:54.834 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:56:54.834 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-10T18:56:54.834 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T18:56:54.834 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T18:56:54.834 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:56:54.834 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:56:54.834 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T18:56:54.834 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T18:56:54.834 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T18:56:54.834 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T18:56:54.834 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T18:56:54.835 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T18:56:54.835 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-10T18:56:54.856 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T18:56:54.856 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T18:56:54.857 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T18:56:54.857 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-10T18:56:54.972 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-10T18:56:54.972 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T18:56:54.972 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T18:56:54.974 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-10T18:56:54.975 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:2 2023-12-10T18:56:54.975 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T18:56:54.975 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:56:54.975 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-10T18:56:54.975 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T18:56:54.975 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T18:56:54.975 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:56:54.975 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:56:54.975 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T18:56:54.975 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T18:56:54.975 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T18:56:54.975 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:56:54.975 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T18:56:54.975 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T18:56:54.975 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-10T18:56:54.975 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:2 2023-12-10T18:56:54.976 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:56:54.976 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:56:54.976 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-10T18:56:54.976 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T18:56:54.976 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T18:56:54.976 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:56:54.976 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:56:54.976 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T18:56:54.976 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T18:56:54.976 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T18:56:54.976 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T18:56:54.976 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T18:56:54.976 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T18:56:54.976 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-10T18:56:54.998 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T18:56:54.999 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T18:56:54.999 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T18:56:54.999 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-10T18:56:55.123 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:56:55.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ status_result= "state": "Replaying" 2023-12-10T18:56:55.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ + grep -i Replaying 2023-12-10T18:56:55.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:echo "state": "Replaying" 2023-12-10T18:56:55.125 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "state": "Replaying" 2023-12-10T18:56:55.126 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=started 2023-12-10T18:56:55.126 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test started = started 2023-12-10T18:56:55.126 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:56:55.126 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-10T18:56:55.126 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T18:56:55.126 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T18:56:55.126 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:56:55.126 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:56:55.127 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = journal ] 2023-12-10T18:56:55.127 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = snapshot ] 2023-12-10T18:56:55.127 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-10T18:56:55.127 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T18:56:55.127 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T18:56:55.127 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:56:55.127 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:56:55.127 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster2-mirror-test.status 2023-12-10T18:56:55.127 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2-mirror-test.status 2023-12-10T18:56:55.127 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T18:56:55.128 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-test.status 2023-12-10T18:56:55.128 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-test.status 2023-12-10T18:56:55.128 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-test.status 2023-12-10T18:56:55.128 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T18:56:55.129 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:56:55.129 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-10T18:56:55.129 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T18:56:55.129 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:56:55.129 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:56:55.129 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-10T18:56:56.722 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Snapshot ID: 21 2023-12-10T18:56:56.742 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.status 2023-12-10T18:56:56.742 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T18:56:56.742 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:56:56.742 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:56:56.743 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-test.status 2023-12-10T18:56:56.743 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-10T18:56:56.743 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T18:56:56.819 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-10T18:56:56.821 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snapshot_id=21 2023-12-10T18:56:56.822 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T18:56:56.822 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.2 2023-12-10T18:56:57.023 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:56:57.023 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:56:57.023 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:56:57.024 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:56:57.024 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:56:57.024 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-10T18:56:57.024 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T18:56:57.096 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T18:56:57.097 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T18:56:57.099 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T18:56:57.099 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T18:56:57.099 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T18:56:57.099 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T18:56:57.099 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 21 = 2023-12-10T18:56:57.100 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.4 2023-12-10T18:56:57.501 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:56:57.501 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:56:57.501 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:56:57.501 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:56:57.501 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:56:57.501 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-10T18:56:57.501 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T18:56:57.573 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T18:56:57.573 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T18:56:57.576 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T18:56:57.576 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T18:56:57.576 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T18:56:57.577 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T18:56:57.577 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 21 = 2023-12-10T18:56:57.577 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.8 2023-12-10T18:56:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:56:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:56:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:56:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:56:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:56:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-10T18:56:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T18:56:58.454 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T18:56:58.454 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T18:56:58.456 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T18:56:58.456 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T18:56:58.456 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T18:56:58.456 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T18:56:58.456 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 21 = 2023-12-10T18:56:58.456 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1.6 2023-12-10T18:57:00.058 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:57:00.059 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:57:00.059 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:57:00.059 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:57:00.059 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:57:00.059 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-10T18:57:00.059 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T18:57:00.144 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T18:57:00.147 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=21 2023-12-10T18:57:00.147 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 21 = 21 2023-12-10T18:57:00.147 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:57:00.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snap_present cluster1 mirror test snap5 2023-12-10T18:57:00.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:57:00.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:57:00.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:57:00.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap5 2023-12-10T18:57:00.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T18:57:00.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T18:57:01.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 info mirror/test@snap5 2023-12-10T18:57:01.212 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'test': 2023-12-10T18:57:01.212 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 512 MiB in 128 objects 2023-12-10T18:57:01.212 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T18:57:01.212 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 10 2023-12-10T18:57:01.212 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 10c0be4f5681 2023-12-10T18:57:01.212 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.10c0be4f5681 2023-12-10T18:57:01.212 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T18:57:01.212 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering, non-primary 2023-12-10T18:57:01.213 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T18:57:01.213 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T18:57:01.213 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 18:52:46 2023 2023-12-10T18:57:01.213 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 18:56:05 2023 2023-12-10T18:57:01.213 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 18:52:46 2023 2023-12-10T18:57:01.213 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: protected: False 2023-12-10T18:57:01.213 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T18:57:01.213 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T18:57:01.213 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: c93b253b-04a0-4ada-b252-fcc724b6756e 2023-12-10T18:57:01.213 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: false 2023-12-10T18:57:01.217 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:57:01.217 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n snap4-clean ] 2023-12-10T18:57:01.217 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror test snap4-clean 2023-12-10T18:57:01.217 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:57:01.217 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:57:01.217 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap4-clean 2023-12-10T18:57:01.217 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T18:57:01.217 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T18:57:01.217 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:57:01.217 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:57:01.218 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T18:57:01.950 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T18:57:01.969 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:57:02.907 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T18:57:02.926 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:57:03.205 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:57:03.340 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:57:03.340 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror test snap5 2023-12-10T18:57:03.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:57:03.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:57:03.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap5 2023-12-10T18:57:03.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T18:57:03.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T18:57:03.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:57:03.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:57:03.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T18:57:04.013 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T18:57:04.026 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:57:05.194 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T18:57:05.215 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:57:05.495 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:57:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:57:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ clean_snap_name=snap5-clean 2023-12-10T18:57:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror test snap5-clean 2023-12-10T18:57:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T18:57:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:57:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:57:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap5-clean 2023-12-10T18:57:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5-clean --debug-rbd=20 --debug-journaler=20 2023-12-10T18:57:06.694 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stress_write_image cluster2 mirror test 2023-12-10T18:57:06.694 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T18:57:06.694 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:57:06.695 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:57:06.695 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-10T18:57:06.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local duration=9 2023-12-10T18:57:06.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T18:57:06.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-10T18:57:06.930 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:100 IOs, 8995840 bytes 2023-12-10T18:57:07.042 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:200 IOs, 9617408 bytes 2023-12-10T18:57:07.276 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:300 IOs, 87491584 bytes 2023-12-10T18:57:07.432 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:400 IOs, 119543808 bytes 2023-12-10T18:57:07.602 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:500 IOs, 162739200 bytes 2023-12-10T18:57:07.706 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.8192454434560602) > weight (0.25) 2023-12-10T18:57:07.706 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.46687095372911447) > weight (0.25) 2023-12-10T18:57:07.706 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.3712802692271432) > weight (0.25) 2023-12-10T18:57:07.706 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.4416025159802772) > weight (0.25) 2023-12-10T18:57:07.706 INFO:tasks.rbd_mirror_thrash:waiting for 64 secs before thrashing 2023-12-10T18:57:07.795 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:600 IOs, 222283776 bytes 2023-12-10T18:57:07.955 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:700 IOs, 273441792 bytes 2023-12-10T18:57:08.073 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:800 IOs, 290034688 bytes 2023-12-10T18:57:08.219 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:900 IOs, 371759104 bytes 2023-12-10T18:57:08.454 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1000 IOs, 405953536 bytes 2023-12-10T18:57:08.711 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1100 IOs, 512798720 bytes 2023-12-10T18:57:08.955 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1200 IOs, 582580224 bytes 2023-12-10T18:57:09.141 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1300 IOs, 638119936 bytes 2023-12-10T18:57:09.297 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1400 IOs, 685553664 bytes 2023-12-10T18:57:09.523 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1500 IOs, 744339456 bytes 2023-12-10T18:57:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1600 IOs, 759921664 bytes 2023-12-10T18:57:09.725 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1700 IOs, 779305984 bytes 2023-12-10T18:57:09.973 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1800 IOs, 841608192 bytes 2023-12-10T18:57:10.089 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1900 IOs, 857715712 bytes 2023-12-10T18:57:10.306 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2000 IOs, 940894208 bytes 2023-12-10T18:57:10.423 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2100 IOs, 941520896 bytes 2023-12-10T18:57:10.624 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2200 IOs, 1006914560 bytes 2023-12-10T18:57:10.757 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2300 IOs, 1038817280 bytes 2023-12-10T18:57:10.885 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2400 IOs, 1070736384 bytes 2023-12-10T18:57:11.075 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2500 IOs, 1129353216 bytes 2023-12-10T18:57:11.217 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2600 IOs, 1161706496 bytes 2023-12-10T18:57:11.357 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2700 IOs, 1198155776 bytes 2023-12-10T18:57:11.440 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2800 IOs, 1203686400 bytes 2023-12-10T18:57:11.571 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2900 IOs, 1246810112 bytes 2023-12-10T18:57:11.689 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3000 IOs, 1247440896 bytes 2023-12-10T18:57:11.830 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3100 IOs, 1282705408 bytes 2023-12-10T18:57:12.017 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3200 IOs, 1338267648 bytes 2023-12-10T18:57:12.141 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3300 IOs, 1368776704 bytes 2023-12-10T18:57:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3400 IOs, 1399449600 bytes 2023-12-10T18:57:12.385 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3500 IOs, 1407982592 bytes 2023-12-10T18:57:12.564 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3600 IOs, 1461816320 bytes 2023-12-10T18:57:12.738 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3700 IOs, 1508590592 bytes 2023-12-10T18:57:12.848 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3800 IOs, 1522407424 bytes 2023-12-10T18:57:12.931 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3900 IOs, 1535632384 bytes 2023-12-10T18:57:13.084 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4000 IOs, 1558840320 bytes 2023-12-10T18:57:13.219 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4100 IOs, 1604329472 bytes 2023-12-10T18:57:13.435 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4200 IOs, 1669669888 bytes 2023-12-10T18:57:13.665 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4300 IOs, 1729916928 bytes 2023-12-10T18:57:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4400 IOs, 1774667776 bytes 2023-12-10T18:57:14.012 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4500 IOs, 1831476224 bytes 2023-12-10T18:57:14.171 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4600 IOs, 1879924736 bytes 2023-12-10T18:57:14.330 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4700 IOs, 1919716352 bytes 2023-12-10T18:57:14.684 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4800 IOs, 2050637824 bytes 2023-12-10T18:57:14.932 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4900 IOs, 2127782912 bytes 2023-12-10T18:57:15.185 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5000 IOs, 2219243520 bytes 2023-12-10T18:57:15.334 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5100 IOs, 2273352704 bytes 2023-12-10T18:57:15.475 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5200 IOs, 2290772992 bytes 2023-12-10T18:57:15.705 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=124 2023-12-10T18:57:15.706 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T18:57:15.706 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 124 -eq 124 ] 2023-12-10T18:57:15.706 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:57:15.706 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-10T18:57:15.706 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:57:15.706 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:57:15.706 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:57:15.706 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state_pattern=up+replaying 2023-12-10T18:57:15.706 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local description_pattern= 2023-12-10T18:57:15.706 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local service_pattern= 2023-12-10T18:57:15.706 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T18:57:16.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-10T18:57:16.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:57:16.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:57:16.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:57:16.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state_pattern=up+replaying 2023-12-10T18:57:16.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local description_pattern= 2023-12-10T18:57:16.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local service_pattern= 2023-12-10T18:57:16.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-test.mirror_status 2023-12-10T18:57:16.704 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-test.mirror_status 2023-12-10T18:57:16.704 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T18:57:16.705 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T18:57:16.706 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ CEPH_ARGS= rbd+ --cluster cluster1 mirror image status mirror/test 2023-12-10T18:57:16.706 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:tee /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T18:57:16.782 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:test: 2023-12-10T18:57:16.782 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: global_id: c93b253b-04a0-4ada-b252-fcc724b6756e 2023-12-10T18:57:16.782 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: state: up+replaying 2023-12-10T18:57:16.782 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: description: replaying, {"bytes_per_second":17691545.6,"bytes_per_snapshot":528019968.0,"last_snapshot_bytes":530746368,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1702234615,"remote_snapshot_timestamp":1702234615,"replay_state":"idle"} 2023-12-10T18:57:16.783 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: service: mirror.2 on smithi076 2023-12-10T18:57:16.783 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: last_update: 2023-12-10 18:57:14 2023-12-10T18:57:16.783 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: peer_sites: 2023-12-10T18:57:16.783 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: name: cluster2 2023-12-10T18:57:16.783 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: state: up+stopped 2023-12-10T18:57:16.783 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: description: local image is primary 2023-12-10T18:57:16.783 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: last_update: 2023-12-10 18:56:48 2023-12-10T18:57:16.792 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T18:57:16.793 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: state: up+replaying 2023-12-10T18:57:16.793 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep ^ description: .* /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T18:57:16.795 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: description: replaying, {"bytes_per_second":17691545.6,"bytes_per_snapshot":528019968.0,"last_snapshot_bytes":530746368,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1702234615,"remote_snapshot_timestamp":1702234615,"replay_state":"idle"} 2023-12-10T18:57:16.795 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T18:57:16.795 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo up+replaying 2023-12-10T18:57:16.796 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep ^up+ 2023-12-10T18:57:16.796 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:up+replaying 2023-12-10T18:57:16.796 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep service: *mirror..* on /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T18:57:16.797 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: service: mirror.2 on smithi076 2023-12-10T18:57:16.798 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T18:57:16.799 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local last_update=2023-12-10 18:57:14 2023-12-10T18:57:16.799 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying 2023-12-10 18:57:14 2023-12-10T18:57:16.799 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:57:16.799 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:57:16.799 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:57:16.799 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state_pattern=up+replaying 2023-12-10T18:57:16.800 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local prev_last_update=2023-12-10 18:57:14 2023-12-10T18:57:16.800 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror.mirror_status 2023-12-10T18:57:16.800 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror.mirror_status 2023-12-10T18:57:16.800 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T18:57:16.801 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror.mirror_status 2023-12-10T18:57:16.801 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-10T18:57:16.850 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: failed to query services: (13) Permission denied 2023-12-10T18:57:16.897 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local last_update state 2023-12-10T18:57:16.897 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2023-12-10T18:57:16.901 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ last_update=2023-12-10 18:57:14 2023-12-10T18:57:16.901 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2023-12-10T18:57:16.904 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ state=up+replaying 2023-12-10T18:57:16.904 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo up+replaying 2023-12-10T18:57:16.904 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep up+replaying 2023-12-10T18:57:16.905 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:up+replaying 2023-12-10T18:57:16.905 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:57:16.906 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:57:16.906 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ snap_name=snap6 2023-12-10T18:57:16.906 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror test snap6 2023-12-10T18:57:16.906 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T18:57:16.906 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:57:16.906 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:57:16.906 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap6 2023-12-10T18:57:16.906 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6 --debug-rbd=20 --debug-journaler=20 2023-12-10T18:57:42.721 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-10T18:57:42.721 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:57:42.722 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:57:42.722 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:57:42.722 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-10T18:57:42.722 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:57:42.722 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:57:42.722 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:57:42.722 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=started 2023-12-10T18:57:42.722 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T18:57:42.722 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T18:57:43.722 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-10T18:57:43.722 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:57:43.722 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:57:43.722 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:57:43.722 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=started 2023-12-10T18:57:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_result 2023-12-10T18:57:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=stopped 2023-12-10T18:57:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-10T18:57:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_instance=cluster1 2023-12-10T18:57:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T18:57:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:57:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance=cluster1 2023-12-10T18:57:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ + local loop_instance 2023-12-10T18:57:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T18:57:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:grep -i state 2023-12-10T18:57:43.725 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T18:57:43.725 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T18:57:43.725 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-10T18:57:43.725 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T18:57:43.725 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T18:57:43.726 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:57:43.726 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T18:57:43.726 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T18:57:43.726 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T18:57:43.726 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:57:43.726 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:57:43.726 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T18:57:43.726 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T18:57:43.726 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T18:57:43.726 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:57:43.726 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T18:57:43.726 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T18:57:43.726 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T18:57:43.726 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T18:57:43.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:57:43.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:57:43.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T18:57:43.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T18:57:43.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T18:57:43.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:57:43.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:57:43.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T18:57:43.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T18:57:43.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T18:57:43.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T18:57:43.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T18:57:43.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T18:57:43.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T18:57:43.747 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T18:57:43.747 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T18:57:43.747 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T18:57:43.747 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-10T18:57:43.857 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-10T18:57:43.857 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T18:57:43.857 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T18:57:43.859 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-10T18:57:43.860 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:1 2023-12-10T18:57:43.860 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T18:57:43.860 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:57:43.860 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-10T18:57:43.860 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T18:57:43.860 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T18:57:43.860 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:57:43.860 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:57:43.860 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T18:57:43.860 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T18:57:43.860 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T18:57:43.860 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:57:43.860 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T18:57:43.860 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T18:57:43.861 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-10T18:57:43.861 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:1 2023-12-10T18:57:43.861 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:57:43.861 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:57:43.861 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-10T18:57:43.861 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T18:57:43.861 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T18:57:43.862 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:57:43.862 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:57:43.862 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T18:57:43.862 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T18:57:43.862 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T18:57:43.862 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T18:57:43.862 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T18:57:43.862 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T18:57:43.863 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-10T18:57:43.885 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T18:57:43.885 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T18:57:43.885 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T18:57:43.885 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-10T18:57:43.986 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-10T18:57:43.986 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T18:57:43.986 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T18:57:43.988 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-10T18:57:43.989 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:2 2023-12-10T18:57:43.989 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T18:57:43.989 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:57:43.989 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-10T18:57:43.989 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T18:57:43.989 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T18:57:43.989 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:57:43.989 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:57:43.989 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T18:57:43.989 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T18:57:43.989 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T18:57:43.989 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:57:43.989 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T18:57:43.989 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T18:57:43.989 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-10T18:57:43.989 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:2 2023-12-10T18:57:43.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:57:43.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:57:43.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-10T18:57:43.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T18:57:43.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T18:57:43.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:57:43.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:57:43.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T18:57:43.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T18:57:43.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T18:57:43.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T18:57:43.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T18:57:43.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T18:57:43.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-10T18:57:44.008 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T18:57:44.008 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T18:57:44.008 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T18:57:44.008 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-10T18:57:44.125 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:57:44.125 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ status_result= "state": "Replaying" 2023-12-10T18:57:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo "state": "Replaying" 2023-12-10T18:57:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep -i Replaying 2023-12-10T18:57:44.127 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "state": "Replaying" 2023-12-10T18:57:44.128 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=started 2023-12-10T18:57:44.128 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test started = started 2023-12-10T18:57:44.128 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:57:44.128 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-10T18:57:44.129 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T18:57:44.129 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T18:57:44.129 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:57:44.129 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:57:44.129 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = journal ] 2023-12-10T18:57:44.129 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = snapshot ] 2023-12-10T18:57:44.129 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-10T18:57:44.129 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T18:57:44.129 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T18:57:44.129 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:57:44.129 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:57:44.129 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster2-mirror-test.status 2023-12-10T18:57:44.129 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2-mirror-test.status 2023-12-10T18:57:44.129 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T18:57:44.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-test.status 2023-12-10T18:57:44.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-test.status 2023-12-10T18:57:44.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-test.status 2023-12-10T18:57:44.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T18:57:44.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:57:44.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-10T18:57:44.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T18:57:44.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:57:44.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:57:44.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-10T18:57:45.719 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Snapshot ID: 25 2023-12-10T18:57:45.745 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.status 2023-12-10T18:57:45.745 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T18:57:45.745 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:57:45.745 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:57:45.745 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-test.status 2023-12-10T18:57:45.745 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-10T18:57:45.746 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T18:57:45.830 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-10T18:57:45.833 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snapshot_id=25 2023-12-10T18:57:45.833 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T18:57:45.833 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.2 2023-12-10T18:57:46.034 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:57:46.034 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:57:46.034 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:57:46.034 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:57:46.034 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:57:46.034 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-10T18:57:46.035 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T18:57:46.109 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T18:57:46.109 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T18:57:46.111 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T18:57:46.111 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T18:57:46.111 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T18:57:46.111 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T18:57:46.111 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 25 = 2023-12-10T18:57:46.111 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.4 2023-12-10T18:57:46.513 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:57:46.513 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:57:46.513 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:57:46.513 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:57:46.513 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:57:46.513 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-10T18:57:46.513 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T18:57:46.586 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T18:57:46.586 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T18:57:46.589 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T18:57:46.589 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T18:57:46.590 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T18:57:46.590 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T18:57:46.590 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 25 = 2023-12-10T18:57:46.590 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.8 2023-12-10T18:57:47.391 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:57:47.391 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:57:47.391 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:57:47.392 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:57:47.392 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:57:47.392 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-10T18:57:47.392 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T18:57:47.706 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T18:57:47.707 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T18:57:47.709 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T18:57:47.709 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T18:57:47.709 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T18:57:47.710 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T18:57:47.710 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 25 = 2023-12-10T18:57:47.710 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1.6 2023-12-10T18:57:49.311 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:57:49.312 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:57:49.312 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:57:49.312 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:57:49.312 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:57:49.312 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-10T18:57:49.312 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T18:57:49.386 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T18:57:49.389 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=25 2023-12-10T18:57:49.389 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 25 = 25 2023-12-10T18:57:49.389 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:57:49.389 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snap_present cluster1 mirror test snap6 2023-12-10T18:57:49.389 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:57:49.389 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:57:49.389 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:57:49.389 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap6 2023-12-10T18:57:49.389 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T18:57:49.389 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T18:57:50.390 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 info mirror/test@snap6 2023-12-10T18:57:50.455 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'test': 2023-12-10T18:57:50.455 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 512 MiB in 128 objects 2023-12-10T18:57:50.455 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T18:57:50.455 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 12 2023-12-10T18:57:50.456 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 10c0be4f5681 2023-12-10T18:57:50.456 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.10c0be4f5681 2023-12-10T18:57:50.456 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T18:57:50.456 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering, non-primary 2023-12-10T18:57:50.456 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T18:57:50.456 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T18:57:50.456 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 18:52:46 2023 2023-12-10T18:57:50.456 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 18:56:05 2023 2023-12-10T18:57:50.456 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 18:52:46 2023 2023-12-10T18:57:50.456 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: protected: False 2023-12-10T18:57:50.456 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T18:57:50.456 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T18:57:50.456 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: c93b253b-04a0-4ada-b252-fcc724b6756e 2023-12-10T18:57:50.456 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: false 2023-12-10T18:57:50.460 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:57:50.460 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n snap5-clean ] 2023-12-10T18:57:50.460 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror test snap5-clean 2023-12-10T18:57:50.460 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:57:50.460 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:57:50.460 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap5-clean 2023-12-10T18:57:50.460 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T18:57:50.460 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T18:57:50.460 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:57:50.460 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:57:50.461 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T18:57:50.614 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-12-10T18:57:50.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-12-10T18:57:50.614 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2023-12-10T18:57:50.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr:2023-12-10T18:57:50.611+0000 7f567919d640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 42400) UID: 1000 2023-12-10T18:57:51.128 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T18:57:51.150 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:57:52.279 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T18:57:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:57:52.580 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:57:52.716 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:57:52.716 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror test snap6 2023-12-10T18:57:52.716 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:57:52.716 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:57:52.716 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap6 2023-12-10T18:57:52.716 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T18:57:52.716 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T18:57:52.716 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:57:52.717 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:57:52.717 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T18:57:53.385 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T18:57:53.405 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:57:54.789 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T18:57:54.814 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:57:55.095 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:57:55.229 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:57:55.230 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ clean_snap_name=snap6-clean 2023-12-10T18:57:55.230 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror test snap6-clean 2023-12-10T18:57:55.230 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T18:57:55.230 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:57:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:57:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap6-clean 2023-12-10T18:57:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6-clean --debug-rbd=20 --debug-journaler=20 2023-12-10T18:57:55.616 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-12-10T18:57:55.616 INFO:teuthology.orchestra.run:waiting for 600 2023-12-10T18:57:55.616 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-12-10T18:57:55.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-10T18:57:55.617 DEBUG:teuthology.orchestra.run.smithi076:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-10T18:57:55.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-10T18:57:55.620 INFO:tasks.rbd_mirror_thrash:waiting for 117 secs before thrashing 2023-12-10T18:57:55.696 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stress_write_image cluster2 mirror test 2023-12-10T18:57:55.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T18:57:55.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:57:55.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:57:55.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-10T18:57:55.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local duration=8 2023-12-10T18:57:55.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T18:57:55.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-10T18:57:56.032 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:100 IOs, 50782208 bytes 2023-12-10T18:57:56.256 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:200 IOs, 115940352 bytes 2023-12-10T18:57:56.398 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:300 IOs, 140567552 bytes 2023-12-10T18:57:56.554 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:400 IOs, 173320192 bytes 2023-12-10T18:57:56.661 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:500 IOs, 188100608 bytes 2023-12-10T18:57:56.799 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:600 IOs, 216770560 bytes 2023-12-10T18:57:56.945 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:700 IOs, 274189312 bytes 2023-12-10T18:57:57.099 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:800 IOs, 293161984 bytes 2023-12-10T18:57:57.179 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:900 IOs, 298051584 bytes 2023-12-10T18:57:57.304 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1000 IOs, 324579328 bytes 2023-12-10T18:57:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1100 IOs, 329356288 bytes 2023-12-10T18:57:57.499 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1200 IOs, 343464960 bytes 2023-12-10T18:57:57.609 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1300 IOs, 398222336 bytes 2023-12-10T18:57:57.850 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1400 IOs, 456768512 bytes 2023-12-10T18:57:58.150 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1500 IOs, 552910848 bytes 2023-12-10T18:57:58.329 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1600 IOs, 604284928 bytes 2023-12-10T18:57:58.406 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1700 IOs, 604902400 bytes 2023-12-10T18:57:58.550 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1800 IOs, 641227776 bytes 2023-12-10T18:57:58.668 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1900 IOs, 656460800 bytes 2023-12-10T18:57:58.768 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2000 IOs, 670733312 bytes 2023-12-10T18:57:58.891 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2100 IOs, 696227840 bytes 2023-12-10T18:57:59.013 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2200 IOs, 721016832 bytes 2023-12-10T18:57:59.193 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2300 IOs, 777333760 bytes 2023-12-10T18:57:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2400 IOs, 829918208 bytes 2023-12-10T18:57:59.520 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2500 IOs, 863179776 bytes 2023-12-10T18:57:59.715 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2600 IOs, 916708352 bytes 2023-12-10T18:57:59.931 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2700 IOs, 980043776 bytes 2023-12-10T18:58:00.086 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2800 IOs, 1015342080 bytes 2023-12-10T18:58:00.248 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2900 IOs, 1071857664 bytes 2023-12-10T18:58:00.539 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3000 IOs, 1151053824 bytes 2023-12-10T18:58:00.716 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3100 IOs, 1192115200 bytes 2023-12-10T18:58:00.925 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3200 IOs, 1256138752 bytes 2023-12-10T18:58:01.111 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3300 IOs, 1311495168 bytes 2023-12-10T18:58:01.314 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3400 IOs, 1378736128 bytes 2023-12-10T18:58:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3500 IOs, 1451845632 bytes 2023-12-10T18:58:01.647 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3600 IOs, 1486130176 bytes 2023-12-10T18:58:01.767 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3700 IOs, 1507268608 bytes 2023-12-10T18:58:01.917 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3800 IOs, 1534140416 bytes 2023-12-10T18:58:02.041 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3900 IOs, 1561573376 bytes 2023-12-10T18:58:02.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4000 IOs, 1575258112 bytes 2023-12-10T18:58:02.270 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4100 IOs, 1604218880 bytes 2023-12-10T18:58:02.409 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4200 IOs, 1640478720 bytes 2023-12-10T18:58:02.629 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4300 IOs, 1689270272 bytes 2023-12-10T18:58:02.721 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4400 IOs, 1695709184 bytes 2023-12-10T18:58:02.838 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4500 IOs, 1715315712 bytes 2023-12-10T18:58:02.922 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4600 IOs, 1735716864 bytes 2023-12-10T18:58:03.141 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4700 IOs, 1792567296 bytes 2023-12-10T18:58:03.315 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4800 IOs, 1837790208 bytes 2023-12-10T18:58:03.407 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4900 IOs, 1842017280 bytes 2023-12-10T18:58:03.549 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5000 IOs, 1874725888 bytes 2023-12-10T18:58:03.635 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5100 IOs, 1875351552 bytes 2023-12-10T18:58:03.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=124 2023-12-10T18:58:03.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T18:58:03.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 124 -eq 124 ] 2023-12-10T18:58:03.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:58:03.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-10T18:58:03.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:58:03.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:58:03.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:58:03.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state_pattern=up+replaying 2023-12-10T18:58:03.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local description_pattern= 2023-12-10T18:58:03.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local service_pattern= 2023-12-10T18:58:03.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T18:58:04.704 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-10T18:58:04.704 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:58:04.704 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:58:04.704 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:58:04.704 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state_pattern=up+replaying 2023-12-10T18:58:04.704 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local description_pattern= 2023-12-10T18:58:04.704 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local service_pattern= 2023-12-10T18:58:04.705 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-test.mirror_status 2023-12-10T18:58:04.705 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-test.mirror_status 2023-12-10T18:58:04.705 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T18:58:04.705 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T18:58:04.705 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2023-12-10T18:58:04.706 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ tee /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T18:58:04.794 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:test: 2023-12-10T18:58:04.795 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: global_id: c93b253b-04a0-4ada-b252-fcc724b6756e 2023-12-10T18:58:04.795 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: state: up+replaying 2023-12-10T18:58:04.795 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":528019968.0,"last_snapshot_bytes":530746368,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1702234615,"remote_snapshot_timestamp":1702234615,"replay_state":"idle"} 2023-12-10T18:58:04.795 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: service: mirror.2 on smithi076 2023-12-10T18:58:04.795 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: last_update: 2023-12-10 18:57:44 2023-12-10T18:58:04.795 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: peer_sites: 2023-12-10T18:58:04.795 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: name: cluster2 2023-12-10T18:58:04.795 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: state: up+stopped 2023-12-10T18:58:04.795 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: description: local image is primary 2023-12-10T18:58:04.795 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: last_update: 2023-12-10 18:57:48 2023-12-10T18:58:04.803 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T18:58:04.805 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: state: up+replaying 2023-12-10T18:58:04.805 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep ^ description: .* /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T18:58:04.806 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":528019968.0,"last_snapshot_bytes":530746368,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1702234615,"remote_snapshot_timestamp":1702234615,"replay_state":"idle"} 2023-12-10T18:58:04.806 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T18:58:04.806 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo up+replaying 2023-12-10T18:58:04.807 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep ^up+ 2023-12-10T18:58:04.808 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:up+replaying 2023-12-10T18:58:04.808 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep service: *mirror..* on /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T18:58:04.809 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: service: mirror.2 on smithi076 2023-12-10T18:58:04.809 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T18:58:04.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local last_update=2023-12-10 18:57:44 2023-12-10T18:58:04.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying 2023-12-10 18:57:44 2023-12-10T18:58:04.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:58:04.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:58:04.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:58:04.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state_pattern=up+replaying 2023-12-10T18:58:04.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local prev_last_update=2023-12-10 18:57:44 2023-12-10T18:58:04.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror.mirror_status 2023-12-10T18:58:04.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror.mirror_status 2023-12-10T18:58:04.813 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T18:58:04.813 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror.mirror_status 2023-12-10T18:58:04.813 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-10T18:58:04.867 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: failed to query services: (13) Permission denied 2023-12-10T18:58:04.917 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local last_update state 2023-12-10T18:58:04.917 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2023-12-10T18:58:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ last_update=2023-12-10 18:57:44 2023-12-10T18:58:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2023-12-10T18:58:04.925 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ state=up+replaying 2023-12-10T18:58:04.925 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo up+replaying 2023-12-10T18:58:04.925 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep up+replaying 2023-12-10T18:58:04.926 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:up+replaying 2023-12-10T18:58:04.927 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:58:04.927 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:58:04.927 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ snap_name=snap7 2023-12-10T18:58:04.927 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror test snap7 2023-12-10T18:58:04.927 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T18:58:04.927 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:58:04.927 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:58:04.927 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap7 2023-12-10T18:58:04.927 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7 --debug-rbd=20 --debug-journaler=20 2023-12-10T18:58:11.707 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2023-12-10T18:58:11.708 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2023-12-10T18:58:11.708 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2023-12-10T18:58:11.709 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi076.stderr:2023-12-10T18:58:11.706+0000 7f1ffdb4c640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 39908) UID: 1000 2023-12-10T18:58:13.709 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2023-12-10T18:58:13.709 INFO:teuthology.orchestra.run:waiting for 600 2023-12-10T18:58:13.709 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2023-12-10T18:58:13.709 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-12-10T18:58:13.710 DEBUG:teuthology.orchestra.run.smithi076:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-12-10T18:58:13.712 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-12-10T18:58:13.712 INFO:tasks.rbd_mirror_thrash:waiting for 98 secs before thrashing 2023-12-10T18:58:34.704 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-10T18:58:34.705 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:58:34.705 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:58:34.705 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:58:34.705 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-10T18:58:34.705 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:58:34.705 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:58:34.705 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:58:34.705 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=started 2023-12-10T18:58:34.705 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T18:58:34.705 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T18:58:35.705 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-10T18:58:35.705 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:58:35.705 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:58:35.705 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:58:35.705 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=started 2023-12-10T18:58:35.705 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_result 2023-12-10T18:58:35.706 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=stopped 2023-12-10T18:58:35.706 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-10T18:58:35.706 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ + local cluster_instance=cluster1 2023-12-10T18:58:35.706 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shiftgrep 2023-12-10T18:58:35.706 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: -i state 2023-12-10T18:58:35.706 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:58:35.707 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance=cluster1 2023-12-10T18:58:35.707 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loop_instance 2023-12-10T18:58:35.707 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T18:58:35.707 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T18:58:35.707 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T18:58:35.709 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-10T18:58:35.709 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T18:58:35.709 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T18:58:35.709 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:58:35.709 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T18:58:35.710 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T18:58:35.710 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T18:58:35.710 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:58:35.710 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:58:35.710 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T18:58:35.710 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T18:58:35.710 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T18:58:35.710 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:58:35.710 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T18:58:35.710 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T18:58:35.711 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T18:58:35.711 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T18:58:35.711 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:58:35.711 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:58:35.711 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T18:58:35.711 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T18:58:35.711 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T18:58:35.711 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:58:35.711 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:58:35.711 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T18:58:35.711 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T18:58:35.711 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T18:58:35.712 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T18:58:35.712 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T18:58:35.712 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T18:58:35.712 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T18:58:35.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T18:58:35.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T18:58:35.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T18:58:35.731 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-10T18:58:35.837 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-10T18:58:35.837 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T18:58:35.837 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T18:58:35.839 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-10T18:58:35.839 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:1 2023-12-10T18:58:35.839 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T18:58:35.839 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:58:35.839 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-10T18:58:35.840 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T18:58:35.840 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T18:58:35.840 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:58:35.840 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:58:35.840 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T18:58:35.840 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T18:58:35.840 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T18:58:35.840 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:58:35.840 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T18:58:35.840 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T18:58:35.840 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-10T18:58:35.840 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:1 2023-12-10T18:58:35.840 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:58:35.840 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:58:35.840 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-10T18:58:35.841 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T18:58:35.841 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T18:58:35.841 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:58:35.841 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:58:35.841 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T18:58:35.841 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T18:58:35.841 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T18:58:35.841 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T18:58:35.841 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T18:58:35.841 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T18:58:35.841 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-10T18:58:35.859 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T18:58:35.859 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T18:58:35.859 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T18:58:35.859 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-10T18:58:35.962 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-10T18:58:35.962 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T18:58:35.963 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T18:58:35.965 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-10T18:58:35.965 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:2 2023-12-10T18:58:35.965 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T18:58:35.965 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:58:35.965 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-10T18:58:35.965 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T18:58:35.965 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T18:58:35.965 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:58:35.965 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:58:35.965 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T18:58:35.965 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T18:58:35.965 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T18:58:35.965 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:58:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T18:58:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T18:58:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-10T18:58:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:2 2023-12-10T18:58:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:58:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:58:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-10T18:58:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T18:58:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T18:58:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:58:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:58:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T18:58:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T18:58:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T18:58:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T18:58:35.967 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T18:58:35.967 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T18:58:35.967 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-10T18:58:35.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T18:58:35.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T18:58:35.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T18:58:35.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-10T18:58:36.102 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:58:36.103 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ status_result= "state": "Replaying" 2023-12-10T18:58:36.103 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo "state": "Replaying" 2023-12-10T18:58:36.103 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep -i Replaying 2023-12-10T18:58:36.104 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "state": "Replaying" 2023-12-10T18:58:36.105 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=started 2023-12-10T18:58:36.105 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test started = started 2023-12-10T18:58:36.105 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:58:36.105 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-10T18:58:36.105 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T18:58:36.106 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T18:58:36.106 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:58:36.106 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:58:36.106 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = journal ] 2023-12-10T18:58:36.106 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = snapshot ] 2023-12-10T18:58:36.106 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-10T18:58:36.106 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T18:58:36.106 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T18:58:36.106 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:58:36.106 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:58:36.106 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster2-mirror-test.status 2023-12-10T18:58:36.106 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2-mirror-test.status 2023-12-10T18:58:36.106 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T18:58:36.107 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-test.status 2023-12-10T18:58:36.107 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-test.status 2023-12-10T18:58:36.107 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-test.status 2023-12-10T18:58:36.107 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T18:58:36.108 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:58:36.108 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-10T18:58:36.108 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T18:58:36.108 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:58:36.108 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:58:36.108 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-10T18:58:37.724 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Snapshot ID: 29 2023-12-10T18:58:37.751 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.status 2023-12-10T18:58:37.751 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T18:58:37.751 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:58:37.752 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:58:37.752 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-test.status 2023-12-10T18:58:37.752 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-10T18:58:37.752 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T18:58:37.833 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-10T18:58:37.835 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snapshot_id=29 2023-12-10T18:58:37.835 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T18:58:37.835 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.2 2023-12-10T18:58:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:58:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:58:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:58:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:58:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:58:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-10T18:58:38.038 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T18:58:38.123 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T18:58:38.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T18:58:38.126 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T18:58:38.126 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T18:58:38.126 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T18:58:38.127 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T18:58:38.127 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 29 = 2023-12-10T18:58:38.127 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.4 2023-12-10T18:58:38.528 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:58:38.528 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:58:38.528 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:58:38.529 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:58:38.529 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:58:38.529 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-10T18:58:38.529 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T18:58:38.607 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T18:58:38.607 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T18:58:38.609 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T18:58:38.609 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T18:58:38.609 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T18:58:38.609 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T18:58:38.610 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 29 = 2023-12-10T18:58:38.610 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.8 2023-12-10T18:58:39.410 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:58:39.411 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:58:39.411 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:58:39.411 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:58:39.411 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:58:39.411 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-10T18:58:39.411 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T18:58:39.549 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T18:58:39.549 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T18:58:39.551 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T18:58:39.551 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T18:58:39.552 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T18:58:39.552 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T18:58:39.552 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 29 = 2023-12-10T18:58:39.552 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1.6 2023-12-10T18:58:40.061 INFO:tasks.ceph.cluster1.osd.0.smithi033.stderr:2023-12-10T18:58:40.055+0000 7f836f941640 -1 reset not still connected to 0x558d96ade410 2023-12-10T18:58:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:58:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:58:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:58:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:58:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:58:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-10T18:58:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T18:58:41.232 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T18:58:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=29 2023-12-10T18:58:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 29 = 29 2023-12-10T18:58:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:58:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snap_present cluster1 mirror test snap7 2023-12-10T18:58:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:58:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:58:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:58:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap7 2023-12-10T18:58:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T18:58:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T18:58:42.236 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 info mirror/test@snap7 2023-12-10T18:58:42.306 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'test': 2023-12-10T18:58:42.306 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 512 MiB in 128 objects 2023-12-10T18:58:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T18:58:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 14 2023-12-10T18:58:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 10c0be4f5681 2023-12-10T18:58:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.10c0be4f5681 2023-12-10T18:58:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T18:58:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering, non-primary 2023-12-10T18:58:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T18:58:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T18:58:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 18:52:46 2023 2023-12-10T18:58:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 18:57:51 2023 2023-12-10T18:58:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 18:52:46 2023 2023-12-10T18:58:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: protected: False 2023-12-10T18:58:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T18:58:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T18:58:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: c93b253b-04a0-4ada-b252-fcc724b6756e 2023-12-10T18:58:42.308 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: false 2023-12-10T18:58:42.311 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:58:42.311 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n snap6-clean ] 2023-12-10T18:58:42.311 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror test snap6-clean 2023-12-10T18:58:42.311 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:58:42.312 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:58:42.312 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap6-clean 2023-12-10T18:58:42.312 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T18:58:42.312 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T18:58:42.312 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:58:42.312 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:58:42.312 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T18:58:42.997 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T18:58:43.018 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:58:43.985 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T18:58:44.001 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:58:44.283 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:58:44.418 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:58:44.418 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror test snap7 2023-12-10T18:58:44.418 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:58:44.418 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:58:44.418 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap7 2023-12-10T18:58:44.418 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T18:58:44.418 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T18:58:44.418 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:58:44.418 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:58:44.419 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T18:58:45.101 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T18:58:45.117 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:58:46.346 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T18:58:46.374 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:58:46.654 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:58:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:58:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ clean_snap_name=snap7-clean 2023-12-10T18:58:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror test snap7-clean 2023-12-10T18:58:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T18:58:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:58:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:58:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap7-clean 2023-12-10T18:58:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7-clean --debug-rbd=20 --debug-journaler=20 2023-12-10T18:58:47.775 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stress_write_image cluster2 mirror test 2023-12-10T18:58:47.775 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T18:58:47.775 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:58:47.775 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:58:47.775 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-10T18:58:47.777 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local duration=5 2023-12-10T18:58:47.777 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T18:58:47.777 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-10T18:58:48.040 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:100 IOs, 25420800 bytes 2023-12-10T18:58:48.188 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:200 IOs, 34248704 bytes 2023-12-10T18:58:48.383 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:300 IOs, 87785472 bytes 2023-12-10T18:58:48.513 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:400 IOs, 120752128 bytes 2023-12-10T18:58:48.680 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:500 IOs, 163073024 bytes 2023-12-10T18:58:48.780 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:600 IOs, 171671552 bytes 2023-12-10T18:58:48.930 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:700 IOs, 213646336 bytes 2023-12-10T18:58:49.107 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:800 IOs, 264137728 bytes 2023-12-10T18:58:49.183 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:900 IOs, 264762368 bytes 2023-12-10T18:58:49.312 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1000 IOs, 288217088 bytes 2023-12-10T18:58:49.418 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1100 IOs, 302012416 bytes 2023-12-10T18:58:49.544 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1200 IOs, 327670784 bytes 2023-12-10T18:58:49.624 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1300 IOs, 330669056 bytes 2023-12-10T18:58:49.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1400 IOs, 344098816 bytes 2023-12-10T18:58:49.985 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1500 IOs, 443250688 bytes 2023-12-10T18:58:50.179 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1600 IOs, 476869632 bytes 2023-12-10T18:58:50.307 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1700 IOs, 492899328 bytes 2023-12-10T18:58:50.461 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1800 IOs, 526319616 bytes 2023-12-10T18:58:50.574 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1900 IOs, 546395136 bytes 2023-12-10T18:58:50.652 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2000 IOs, 547026944 bytes 2023-12-10T18:58:50.827 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2100 IOs, 590270464 bytes 2023-12-10T18:58:51.002 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2200 IOs, 638562304 bytes 2023-12-10T18:58:51.132 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2300 IOs, 668384256 bytes 2023-12-10T18:58:51.229 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2400 IOs, 669005824 bytes 2023-12-10T18:58:51.366 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2500 IOs, 693866496 bytes 2023-12-10T18:58:51.535 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2600 IOs, 744515584 bytes 2023-12-10T18:58:51.696 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2700 IOs, 808477696 bytes 2023-12-10T18:58:51.909 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2800 IOs, 856348672 bytes 2023-12-10T18:58:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2900 IOs, 903696384 bytes 2023-12-10T18:58:52.281 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3000 IOs, 968131584 bytes 2023-12-10T18:58:52.357 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3100 IOs, 968743936 bytes 2023-12-10T18:58:52.582 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3200 IOs, 1046131712 bytes 2023-12-10T18:58:52.743 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3300 IOs, 1099878400 bytes 2023-12-10T18:58:52.780 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=124 2023-12-10T18:58:52.780 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T18:58:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 124 -eq 124 ] 2023-12-10T18:58:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:58:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-10T18:58:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:58:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:58:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:58:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state_pattern=up+replaying 2023-12-10T18:58:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local description_pattern= 2023-12-10T18:58:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local service_pattern= 2023-12-10T18:58:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T18:58:53.782 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-10T18:58:53.782 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:58:53.782 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:58:53.782 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:58:53.782 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state_pattern=up+replaying 2023-12-10T18:58:53.782 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local description_pattern= 2023-12-10T18:58:53.782 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local service_pattern= 2023-12-10T18:58:53.782 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-test.mirror_status 2023-12-10T18:58:53.783 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-test.mirror_status 2023-12-10T18:58:53.783 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T18:58:53.784 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T18:58:53.784 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2023-12-10T18:58:53.784 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ tee /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T18:58:53.866 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:test: 2023-12-10T18:58:53.866 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: global_id: c93b253b-04a0-4ada-b252-fcc724b6756e 2023-12-10T18:58:53.866 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: state: up+replaying 2023-12-10T18:58:53.866 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: description: replaying, {"bytes_per_second":16939758.93,"bytes_per_snapshot":521422848.0,"last_snapshot_bytes":508192768,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1702234716,"remote_snapshot_timestamp":1702234716,"replay_state":"idle"} 2023-12-10T18:58:53.866 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: service: mirror.2 on smithi076 2023-12-10T18:58:53.866 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: last_update: 2023-12-10 18:58:44 2023-12-10T18:58:53.866 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: peer_sites: 2023-12-10T18:58:53.867 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: name: cluster2 2023-12-10T18:58:53.867 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: state: up+stopped 2023-12-10T18:58:53.867 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: description: local image is primary 2023-12-10T18:58:53.867 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: last_update: 2023-12-10 18:58:48 2023-12-10T18:58:53.876 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T18:58:53.877 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: state: up+replaying 2023-12-10T18:58:53.877 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep ^ description: .* /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T18:58:53.878 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: description: replaying, {"bytes_per_second":16939758.93,"bytes_per_snapshot":521422848.0,"last_snapshot_bytes":508192768,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1702234716,"remote_snapshot_timestamp":1702234716,"replay_state":"idle"} 2023-12-10T18:58:53.879 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T18:58:53.879 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo up+replaying 2023-12-10T18:58:53.879 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep ^up+ 2023-12-10T18:58:53.879 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:up+replaying 2023-12-10T18:58:53.879 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep service: *mirror..* on /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T18:58:53.880 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: service: mirror.2 on smithi076 2023-12-10T18:58:53.880 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T18:58:53.882 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local last_update=2023-12-10 18:58:44 2023-12-10T18:58:53.882 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying 2023-12-10 18:58:44 2023-12-10T18:58:53.882 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:58:53.883 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:58:53.883 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:58:53.883 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state_pattern=up+replaying 2023-12-10T18:58:53.883 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local prev_last_update=2023-12-10 18:58:44 2023-12-10T18:58:53.883 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror.mirror_status 2023-12-10T18:58:53.883 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror.mirror_status 2023-12-10T18:58:53.883 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T18:58:53.883 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror.mirror_status 2023-12-10T18:58:53.884 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-10T18:58:53.937 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: failed to query services: (13) Permission denied 2023-12-10T18:58:53.986 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local last_update state 2023-12-10T18:58:53.986 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2023-12-10T18:58:53.989 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ last_update=2023-12-10 18:58:44 2023-12-10T18:58:53.989 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2023-12-10T18:58:53.992 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ state=up+replaying 2023-12-10T18:58:53.992 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo up+replaying 2023-12-10T18:58:53.992 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep up+replaying 2023-12-10T18:58:53.993 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:up+replaying 2023-12-10T18:58:53.994 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:58:53.994 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:58:53.994 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ snap_name=snap8 2023-12-10T18:58:53.994 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror test snap8 2023-12-10T18:58:53.994 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T18:58:53.994 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:58:53.994 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:58:53.994 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap8 2023-12-10T18:58:53.995 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8 --debug-rbd=20 --debug-journaler=20 2023-12-10T18:59:22.734 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-10T18:59:22.735 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:59:22.735 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:59:22.735 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:59:22.735 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-10T18:59:22.735 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:59:22.735 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:59:22.735 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:59:22.735 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=started 2023-12-10T18:59:22.735 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T18:59:22.735 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T18:59:23.735 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-10T18:59:23.735 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:59:23.735 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:59:23.735 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:59:23.735 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=started 2023-12-10T18:59:23.735 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_result 2023-12-10T18:59:23.736 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=stopped 2023-12-10T18:59:23.736 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-10T18:59:23.736 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_instance=cluster1 2023-12-10T18:59:23.736 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T18:59:23.737 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local+ cluster=cluster1 2023-12-10T18:59:23.737 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance=cluster1 2023-12-10T18:59:23.737 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loop_instance 2023-12-10T18:59:23.737 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T18:59:23.737 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:grep -i state 2023-12-10T18:59:23.737 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T18:59:23.737 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T18:59:23.739 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-10T18:59:23.739 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T18:59:23.739 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T18:59:23.739 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:59:23.739 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T18:59:23.740 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T18:59:23.740 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T18:59:23.740 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:59:23.740 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:59:23.740 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T18:59:23.740 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T18:59:23.740 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T18:59:23.740 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:59:23.740 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T18:59:23.740 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T18:59:23.740 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T18:59:23.740 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T18:59:23.741 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:59:23.741 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:59:23.741 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T18:59:23.741 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T18:59:23.741 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T18:59:23.741 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:59:23.741 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:59:23.741 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T18:59:23.741 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T18:59:23.741 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T18:59:23.741 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T18:59:23.741 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T18:59:23.741 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T18:59:23.741 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T18:59:23.761 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T18:59:23.761 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T18:59:23.761 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T18:59:23.761 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-10T18:59:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-10T18:59:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T18:59:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T18:59:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-10T18:59:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:1 2023-12-10T18:59:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T18:59:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:59:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-10T18:59:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T18:59:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T18:59:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:59:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:59:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T18:59:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T18:59:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T18:59:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:59:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T18:59:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T18:59:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-10T18:59:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:1 2023-12-10T18:59:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:59:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:59:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-10T18:59:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T18:59:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T18:59:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:59:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:59:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T18:59:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T18:59:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T18:59:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T18:59:23.879 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T18:59:23.879 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T18:59:23.879 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-10T18:59:23.894 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T18:59:23.894 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T18:59:23.894 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T18:59:23.894 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-10T18:59:23.991 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-10T18:59:23.991 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T18:59:23.991 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T18:59:23.995 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-10T18:59:23.996 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:2 2023-12-10T18:59:23.996 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T18:59:23.996 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:59:23.996 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-10T18:59:23.996 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T18:59:23.996 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T18:59:23.996 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:59:23.996 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:59:23.996 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T18:59:23.996 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T18:59:23.996 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T18:59:23.996 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:59:23.996 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T18:59:23.996 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T18:59:23.996 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-10T18:59:23.996 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:2 2023-12-10T18:59:23.997 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:59:23.997 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T18:59:23.997 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-10T18:59:23.997 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T18:59:23.997 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T18:59:23.997 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T18:59:23.997 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T18:59:23.997 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T18:59:23.997 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T18:59:23.997 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T18:59:23.997 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T18:59:23.997 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T18:59:23.997 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T18:59:23.997 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-10T18:59:24.017 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T18:59:24.017 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T18:59:24.017 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T18:59:24.017 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-10T18:59:24.136 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:59:24.137 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ status_result= "state": "Replaying" 2023-12-10T18:59:24.137 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo "state": "Replaying" 2023-12-10T18:59:24.137 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep -i Replaying 2023-12-10T18:59:24.138 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "state": "Replaying" 2023-12-10T18:59:24.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=started 2023-12-10T18:59:24.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test started = started 2023-12-10T18:59:24.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:59:24.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-10T18:59:24.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T18:59:24.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T18:59:24.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:59:24.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:59:24.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = journal ] 2023-12-10T18:59:24.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = snapshot ] 2023-12-10T18:59:24.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-10T18:59:24.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T18:59:24.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T18:59:24.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:59:24.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:59:24.141 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster2-mirror-test.status 2023-12-10T18:59:24.141 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2-mirror-test.status 2023-12-10T18:59:24.141 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T18:59:24.142 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-test.status 2023-12-10T18:59:24.142 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-test.status 2023-12-10T18:59:24.142 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-test.status 2023-12-10T18:59:24.142 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T18:59:24.142 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:59:24.142 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-10T18:59:24.142 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T18:59:24.143 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:59:24.143 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:59:24.143 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-10T18:59:25.752 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Snapshot ID: 33 2023-12-10T18:59:25.768 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.status 2023-12-10T18:59:25.768 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T18:59:25.768 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:59:25.768 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:59:25.768 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-test.status 2023-12-10T18:59:25.769 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T18:59:25.772 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-10T18:59:25.854 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-10T18:59:25.857 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snapshot_id=33 2023-12-10T18:59:25.857 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T18:59:25.857 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.2 2023-12-10T18:59:26.058 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:59:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:59:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:59:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:59:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:59:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-10T18:59:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T18:59:26.136 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T18:59:26.136 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T18:59:26.138 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T18:59:26.139 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T18:59:26.139 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T18:59:26.139 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T18:59:26.139 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 33 = 2023-12-10T18:59:26.139 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.4 2023-12-10T18:59:26.541 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:59:26.541 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:59:26.541 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:59:26.541 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:59:26.541 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:59:26.542 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-10T18:59:26.542 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T18:59:26.619 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T18:59:26.619 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T18:59:26.622 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T18:59:26.623 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T18:59:26.623 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T18:59:26.624 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T18:59:26.624 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 33 = 2023-12-10T18:59:26.624 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.8 2023-12-10T18:59:27.424 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:59:27.424 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:59:27.424 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:59:27.425 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:59:27.425 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:59:27.425 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd+ --cluster cluster1 snap list --all mirror/test --format xml 2023-12-10T18:59:27.425 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T18:59:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T18:59:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T18:59:27.578 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T18:59:27.578 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T18:59:27.578 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T18:59:27.579 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T18:59:27.579 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 33 = 2023-12-10T18:59:27.579 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1.6 2023-12-10T18:59:29.180 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:59:29.180 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:59:29.180 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:59:29.180 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:59:29.180 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T18:59:29.180 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-10T18:59:29.181 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T18:59:29.256 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T18:59:29.260 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=33 2023-12-10T18:59:29.260 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 33 = 33 2023-12-10T18:59:29.260 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:59:29.260 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snap_present cluster1 mirror test snap8 2023-12-10T18:59:29.260 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:59:29.260 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:59:29.260 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:59:29.260 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap8 2023-12-10T18:59:29.260 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T18:59:29.260 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T18:59:30.261 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 info mirror/test@snap8 2023-12-10T18:59:30.331 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'test': 2023-12-10T18:59:30.332 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 512 MiB in 128 objects 2023-12-10T18:59:30.332 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T18:59:30.332 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 16 2023-12-10T18:59:30.332 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 10c0be4f5681 2023-12-10T18:59:30.332 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.10c0be4f5681 2023-12-10T18:59:30.332 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T18:59:30.332 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering, non-primary 2023-12-10T18:59:30.332 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T18:59:30.332 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T18:59:30.332 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 18:52:46 2023 2023-12-10T18:59:30.332 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 18:57:51 2023 2023-12-10T18:59:30.332 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 18:52:46 2023 2023-12-10T18:59:30.333 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: protected: False 2023-12-10T18:59:30.333 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T18:59:30.333 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T18:59:30.333 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: c93b253b-04a0-4ada-b252-fcc724b6756e 2023-12-10T18:59:30.333 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: false 2023-12-10T18:59:30.336 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:59:30.336 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n snap7-clean ] 2023-12-10T18:59:30.336 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror test snap7-clean 2023-12-10T18:59:30.336 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:59:30.336 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:59:30.336 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap7-clean 2023-12-10T18:59:30.336 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T18:59:30.337 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T18:59:30.337 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:59:30.337 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:59:30.337 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T18:59:31.042 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T18:59:31.054 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:59:32.151 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T18:59:32.174 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:59:32.452 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:59:32.590 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:59:32.590 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror test snap8 2023-12-10T18:59:32.590 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:59:32.590 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:59:32.590 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap8 2023-12-10T18:59:32.590 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T18:59:32.590 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T18:59:32.590 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:59:32.590 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:59:32.591 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T18:59:33.340 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T18:59:33.356 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:59:34.476 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T18:59:34.497 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:59:34.776 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T18:59:34.911 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:59:34.911 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ clean_snap_name=snap8-clean 2023-12-10T18:59:34.911 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror test snap8-clean 2023-12-10T18:59:34.911 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T18:59:34.911 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:59:34.911 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:59:34.911 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap8-clean 2023-12-10T18:59:34.911 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8-clean --debug-rbd=20 --debug-journaler=20 2023-12-10T18:59:35.790 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stress_write_image cluster2 mirror test 2023-12-10T18:59:35.790 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T18:59:35.790 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:59:35.790 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:59:35.790 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-10T18:59:35.792 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local duration=10 2023-12-10T18:59:35.792 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T18:59:35.792 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-10T18:59:36.094 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:100 IOs, 43708416 bytes 2023-12-10T18:59:36.292 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:200 IOs, 83454976 bytes 2023-12-10T18:59:36.515 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:300 IOs, 141012992 bytes 2023-12-10T18:59:36.694 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:400 IOs, 179529728 bytes 2023-12-10T18:59:36.933 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:500 IOs, 241072128 bytes 2023-12-10T18:59:37.220 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:600 IOs, 351256576 bytes 2023-12-10T18:59:37.430 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:700 IOs, 418028544 bytes 2023-12-10T18:59:37.632 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:800 IOs, 486509568 bytes 2023-12-10T18:59:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:900 IOs, 513412096 bytes 2023-12-10T18:59:37.922 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1000 IOs, 560596992 bytes 2023-12-10T18:59:38.121 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1100 IOs, 618646528 bytes 2023-12-10T18:59:38.295 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1200 IOs, 662418432 bytes 2023-12-10T18:59:38.484 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1300 IOs, 716441600 bytes 2023-12-10T18:59:38.758 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1400 IOs, 809756672 bytes 2023-12-10T18:59:38.895 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1500 IOs, 838697984 bytes 2023-12-10T18:59:39.100 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1600 IOs, 897459200 bytes 2023-12-10T18:59:39.229 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1700 IOs, 923130880 bytes 2023-12-10T18:59:39.481 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1800 IOs, 1006807040 bytes 2023-12-10T18:59:39.644 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1900 IOs, 1042542592 bytes 2023-12-10T18:59:39.789 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2000 IOs, 1076895744 bytes 2023-12-10T18:59:39.999 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2100 IOs, 1136211968 bytes 2023-12-10T18:59:40.123 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2200 IOs, 1164409856 bytes 2023-12-10T18:59:40.323 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2300 IOs, 1219348480 bytes 2023-12-10T18:59:40.566 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2400 IOs, 1309109248 bytes 2023-12-10T18:59:40.647 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2500 IOs, 1309720576 bytes 2023-12-10T18:59:40.784 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2600 IOs, 1347075072 bytes 2023-12-10T18:59:40.937 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2700 IOs, 1371164672 bytes 2023-12-10T18:59:41.072 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2800 IOs, 1396234240 bytes 2023-12-10T18:59:41.186 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2900 IOs, 1415913472 bytes 2023-12-10T18:59:41.316 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3000 IOs, 1445063680 bytes 2023-12-10T18:59:41.433 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3100 IOs, 1453092864 bytes 2023-12-10T18:59:41.531 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3200 IOs, 1465879552 bytes 2023-12-10T18:59:41.789 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3300 IOs, 1547781120 bytes 2023-12-10T18:59:41.962 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3400 IOs, 1586577408 bytes 2023-12-10T18:59:42.108 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3500 IOs, 1622020096 bytes 2023-12-10T18:59:42.181 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3600 IOs, 1622630400 bytes 2023-12-10T18:59:42.417 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3700 IOs, 1691453440 bytes 2023-12-10T18:59:42.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3800 IOs, 1725996032 bytes 2023-12-10T18:59:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3900 IOs, 1726589952 bytes 2023-12-10T18:59:42.829 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4000 IOs, 1782090752 bytes 2023-12-10T18:59:42.987 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4100 IOs, 1817276416 bytes 2023-12-10T18:59:43.097 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4200 IOs, 1832031232 bytes 2023-12-10T18:59:43.229 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4300 IOs, 1857089536 bytes 2023-12-10T18:59:43.369 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4400 IOs, 1889094656 bytes 2023-12-10T18:59:43.514 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4500 IOs, 1944720384 bytes 2023-12-10T18:59:43.651 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4600 IOs, 1957956608 bytes 2023-12-10T18:59:43.788 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4700 IOs, 1982134272 bytes 2023-12-10T18:59:43.876 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4800 IOs, 1982744576 bytes 2023-12-10T18:59:44.048 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4900 IOs, 2029964288 bytes 2023-12-10T18:59:44.152 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5000 IOs, 2042481664 bytes 2023-12-10T18:59:44.260 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5100 IOs, 2057608192 bytes 2023-12-10T18:59:44.449 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5200 IOs, 2118156288 bytes 2023-12-10T18:59:44.704 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5300 IOs, 2190061568 bytes 2023-12-10T18:59:44.911 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5400 IOs, 2276232192 bytes 2023-12-10T18:59:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5500 IOs, 2298088448 bytes 2023-12-10T18:59:45.188 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5600 IOs, 2308729856 bytes 2023-12-10T18:59:45.333 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5700 IOs, 2360936448 bytes 2023-12-10T18:59:45.552 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5800 IOs, 2413945856 bytes 2023-12-10T18:59:45.709 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5900 IOs, 2454704128 bytes 2023-12-10T18:59:45.797 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=124 2023-12-10T18:59:45.797 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T18:59:45.797 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 124 -eq 124 ] 2023-12-10T18:59:45.797 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:59:45.797 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-10T18:59:45.797 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:59:45.798 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:59:45.798 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:59:45.798 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state_pattern=up+replaying 2023-12-10T18:59:45.798 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local description_pattern= 2023-12-10T18:59:45.798 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local service_pattern= 2023-12-10T18:59:45.798 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T18:59:46.798 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-10T18:59:46.798 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:59:46.798 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:59:46.798 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:59:46.798 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state_pattern=up+replaying 2023-12-10T18:59:46.799 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local description_pattern= 2023-12-10T18:59:46.799 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local service_pattern= 2023-12-10T18:59:46.799 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-test.mirror_status 2023-12-10T18:59:46.799 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-test.mirror_status 2023-12-10T18:59:46.799 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T18:59:46.800 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T18:59:46.801 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2023-12-10T18:59:46.801 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ tee /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T18:59:46.886 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:test: 2023-12-10T18:59:46.886 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: global_id: c93b253b-04a0-4ada-b252-fcc724b6756e 2023-12-10T18:59:46.886 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: state: up+replaying 2023-12-10T18:59:46.886 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: description: replaying, {"bytes_per_second":16262451.2,"bytes_per_snapshot":498033152.0,"last_snapshot_bytes":487873536,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1702234764,"remote_snapshot_timestamp":1702234764,"replay_state":"idle"} 2023-12-10T18:59:46.886 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: service: mirror.2 on smithi076 2023-12-10T18:59:46.886 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: last_update: 2023-12-10 18:59:44 2023-12-10T18:59:46.886 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: peer_sites: 2023-12-10T18:59:46.886 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: name: cluster2 2023-12-10T18:59:46.886 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: state: up+stopped 2023-12-10T18:59:46.886 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: description: local image is primary 2023-12-10T18:59:46.886 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: last_update: 2023-12-10 18:59:18 2023-12-10T18:59:46.935 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T18:59:46.936 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: state: up+replaying 2023-12-10T18:59:46.937 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep ^ description: .* /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T18:59:46.938 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: description: replaying, {"bytes_per_second":16262451.2,"bytes_per_snapshot":498033152.0,"last_snapshot_bytes":487873536,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1702234764,"remote_snapshot_timestamp":1702234764,"replay_state":"idle"} 2023-12-10T18:59:46.938 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T18:59:46.939 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo up+replaying 2023-12-10T18:59:46.939 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep ^up+ 2023-12-10T18:59:46.940 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:up+replaying 2023-12-10T18:59:46.940 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep service: *mirror..* on /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T18:59:46.941 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: service: mirror.2 on smithi076 2023-12-10T18:59:46.942 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T18:59:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local last_update=2023-12-10 18:59:44 2023-12-10T18:59:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying 2023-12-10 18:59:44 2023-12-10T18:59:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T18:59:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:59:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:59:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state_pattern=up+replaying 2023-12-10T18:59:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local prev_last_update=2023-12-10 18:59:44 2023-12-10T18:59:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror.mirror_status 2023-12-10T18:59:46.945 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror.mirror_status 2023-12-10T18:59:46.945 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T18:59:46.945 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror.mirror_status 2023-12-10T18:59:46.945 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-10T18:59:46.996 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: failed to query services: (13) Permission denied 2023-12-10T18:59:47.047 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local last_update state 2023-12-10T18:59:47.048 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2023-12-10T18:59:47.051 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ last_update=2023-12-10 18:59:44 2023-12-10T18:59:47.051 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2023-12-10T18:59:47.053 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ state=up+replaying 2023-12-10T18:59:47.054 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo up+replaying 2023-12-10T18:59:47.054 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep up+replaying 2023-12-10T18:59:47.054 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:up+replaying 2023-12-10T18:59:47.055 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:59:47.055 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T18:59:47.055 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ snap_name=snap9 2023-12-10T18:59:47.055 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror test snap9 2023-12-10T18:59:47.055 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T18:59:47.056 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T18:59:47.056 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T18:59:47.056 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap9 2023-12-10T18:59:47.056 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9 --debug-rbd=20 --debug-journaler=20 2023-12-10T18:59:51.714 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5700783171108671) > weight (0.25) 2023-12-10T18:59:51.714 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.44309926822469625) > weight (0.25) 2023-12-10T18:59:51.714 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.4454248912351326) > weight (0.25) 2023-12-10T18:59:51.714 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.6473633847795972) > weight (0.25) 2023-12-10T18:59:51.714 INFO:tasks.rbd_mirror_thrash:waiting for 77 secs before thrashing 2023-12-10T18:59:52.620 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-12-10T18:59:52.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-12-10T18:59:52.621 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2023-12-10T18:59:52.621 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-12-10T18:59:52.621 INFO:teuthology.orchestra.run:waiting for 600 2023-12-10T18:59:52.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr:2023-12-10T18:59:52.621+0000 7f3f5ccd9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 44606) UID: 1000 2023-12-10T18:59:58.723 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-12-10T18:59:58.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-10T18:59:58.723 DEBUG:teuthology.orchestra.run.smithi076:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-10T18:59:58.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-10T18:59:58.726 INFO:tasks.rbd_mirror_thrash:waiting for 99 secs before thrashing 2023-12-10T19:00:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-10T19:00:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:00:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:00:13.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:00:13.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-10T19:00:13.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:00:13.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:00:13.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:00:13.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=started 2023-12-10T19:00:13.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:00:13.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:00:14.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-10T19:00:14.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:00:14.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:00:14.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:00:14.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=started 2023-12-10T19:00:14.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_result 2023-12-10T19:00:14.342 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=stopped 2023-12-10T19:00:14.342 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-10T19:00:14.342 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_instance=cluster1 2023-12-10T19:00:14.342 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:00:14.342 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:00:14.342 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance=cluster1 2023-12-10T19:00:14.342 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loop_instance 2023-12-10T19:00:14.343 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ + sleep 0 2023-12-10T19:00:14.343 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:grep -i state 2023-12-10T19:00:14.344 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T19:00:14.344 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T19:00:14.344 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-10T19:00:14.344 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T19:00:14.344 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:00:14.344 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:00:14.344 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T19:00:14.344 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:00:14.344 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:00:14.344 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:00:14.345 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:00:14.345 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:00:14.345 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:00:14.345 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:00:14.345 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:00:14.345 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:00:14.345 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:00:14.345 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T19:00:14.345 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T19:00:14.345 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:00:14.345 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:00:14.345 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T19:00:14.345 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:00:14.345 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:00:14.345 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:00:14.345 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:00:14.346 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:00:14.346 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:00:14.346 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:00:14.346 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:00:14.346 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:00:14.346 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:00:14.346 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T19:00:14.371 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:00:14.371 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:00:14.371 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:00:14.371 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-10T19:00:14.476 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-10T19:00:14.476 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:00:14.476 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:00:14.478 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-10T19:00:14.478 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:1 2023-12-10T19:00:14.478 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:00:14.478 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:00:14.478 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-10T19:00:14.478 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:00:14.478 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:00:14.478 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:00:14.479 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:00:14.479 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:00:14.479 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:00:14.479 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:00:14.479 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:00:14.479 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:00:14.479 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:00:14.479 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-10T19:00:14.479 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:1 2023-12-10T19:00:14.479 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:00:14.479 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:00:14.479 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-10T19:00:14.479 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:00:14.479 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:00:14.479 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:00:14.479 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:00:14.480 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:00:14.480 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:00:14.480 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:00:14.480 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:00:14.480 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:00:14.481 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:00:14.481 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-10T19:00:14.504 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:00:14.504 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:00:14.504 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:00:14.504 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-10T19:00:14.605 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-10T19:00:14.605 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:00:14.605 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:00:14.608 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-10T19:00:14.608 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:2 2023-12-10T19:00:14.608 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:00:14.608 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:00:14.609 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-10T19:00:14.609 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:00:14.609 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:00:14.609 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:00:14.609 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:00:14.609 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:00:14.609 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:00:14.609 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:00:14.609 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:00:14.609 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:00:14.609 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:00:14.609 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-10T19:00:14.609 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:2 2023-12-10T19:00:14.609 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:00:14.609 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:00:14.610 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-10T19:00:14.610 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:00:14.610 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:00:14.610 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:00:14.610 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:00:14.610 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:00:14.610 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:00:14.610 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:00:14.610 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:00:14.610 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:00:14.610 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:00:14.610 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-10T19:00:14.626 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:00:14.627 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:00:14.627 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:00:14.627 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-10T19:00:14.737 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:00:14.738 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ status_result= "state": "Replaying" 2023-12-10T19:00:14.738 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo "state": "Replaying" 2023-12-10T19:00:14.738 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep -i Replaying 2023-12-10T19:00:14.740 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "state": "Replaying" 2023-12-10T19:00:14.741 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=started 2023-12-10T19:00:14.741 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test started = started 2023-12-10T19:00:14.741 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:00:14.741 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-10T19:00:14.741 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:00:14.741 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:00:14.741 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:00:14.741 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:00:14.741 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = journal ] 2023-12-10T19:00:14.741 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = snapshot ] 2023-12-10T19:00:14.742 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-10T19:00:14.742 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:00:14.742 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:00:14.742 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:00:14.742 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:00:14.742 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster2-mirror-test.status 2023-12-10T19:00:14.742 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2-mirror-test.status 2023-12-10T19:00:14.742 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:00:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-test.status 2023-12-10T19:00:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-test.status 2023-12-10T19:00:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-test.status 2023-12-10T19:00:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:00:14.745 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T19:00:14.745 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-10T19:00:14.745 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:00:14.745 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:00:14.745 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:00:14.745 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-10T19:00:16.776 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Snapshot ID: 37 2023-12-10T19:00:16.792 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.status 2023-12-10T19:00:16.793 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:00:16.793 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:00:16.793 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:00:16.793 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-test.status 2023-12-10T19:00:16.793 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:00:16.793 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-10T19:00:16.879 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-10T19:00:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snapshot_id=37 2023-12-10T19:00:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:00:16.882 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.2 2023-12-10T19:00:17.082 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T19:00:17.083 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:00:17.083 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:00:17.083 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:00:17.083 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T19:00:17.083 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-10T19:00:17.083 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:00:17.165 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:00:17.166 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:00:17.168 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:00:17.168 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:00:17.168 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:00:17.168 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:00:17.169 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 37 = 2023-12-10T19:00:17.169 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.4 2023-12-10T19:00:17.570 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T19:00:17.570 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:00:17.571 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:00:17.571 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:00:17.571 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T19:00:17.571 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-10T19:00:17.571 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:00:17.653 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:00:17.653 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:00:17.655 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:00:17.655 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:00:17.655 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:00:17.655 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:00:17.655 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 37 = 2023-12-10T19:00:17.655 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.8 2023-12-10T19:00:18.457 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T19:00:18.457 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:00:18.457 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:00:18.457 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:00:18.457 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T19:00:18.457 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-10T19:00:18.458 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:00:18.595 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:00:18.595 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:00:18.598 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:00:18.598 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:00:18.599 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:00:18.599 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:00:18.599 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 37 = 2023-12-10T19:00:18.599 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1.6 2023-12-10T19:00:20.200 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T19:00:20.201 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:00:20.201 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:00:20.201 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:00:20.201 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T19:00:20.201 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-10T19:00:20.201 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:00:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:00:20.282 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=37 2023-12-10T19:00:20.283 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 37 = 37 2023-12-10T19:00:20.283 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:00:20.283 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snap_present cluster1 mirror test snap9 2023-12-10T19:00:20.283 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:00:20.283 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:00:20.283 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:00:20.283 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap9 2023-12-10T19:00:20.283 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:00:20.283 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:00:21.283 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 info mirror/test@snap9 2023-12-10T19:00:21.358 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'test': 2023-12-10T19:00:21.358 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 512 MiB in 128 objects 2023-12-10T19:00:21.358 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:00:21.358 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 18 2023-12-10T19:00:21.358 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 10c0be4f5681 2023-12-10T19:00:21.358 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.10c0be4f5681 2023-12-10T19:00:21.358 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:00:21.358 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering, non-primary 2023-12-10T19:00:21.358 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:00:21.358 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:00:21.359 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 18:52:46 2023 2023-12-10T19:00:21.359 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 18:59:31 2023 2023-12-10T19:00:21.359 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 18:52:46 2023 2023-12-10T19:00:21.359 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: protected: False 2023-12-10T19:00:21.359 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:00:21.359 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:00:21.359 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: c93b253b-04a0-4ada-b252-fcc724b6756e 2023-12-10T19:00:21.359 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: false 2023-12-10T19:00:21.364 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:00:21.364 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n snap8-clean ] 2023-12-10T19:00:21.364 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror test snap8-clean 2023-12-10T19:00:21.364 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:00:21.364 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:00:21.365 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap8-clean 2023-12-10T19:00:21.365 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:00:21.365 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T19:00:21.365 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:00:21.365 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:00:21.366 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T19:00:22.083 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T19:00:22.097 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:00:23.322 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T19:00:23.343 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:00:23.623 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:00:23.762 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:00:23.762 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror test snap9 2023-12-10T19:00:23.762 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:00:23.762 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:00:23.762 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap9 2023-12-10T19:00:23.762 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:00:23.762 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T19:00:23.762 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:00:23.762 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:00:23.763 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T19:00:24.437 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T19:00:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:00:25.578 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T19:00:25.601 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:00:25.881 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:00:26.018 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:00:26.018 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ clean_snap_name=snap9-clean 2023-12-10T19:00:26.018 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror test snap9-clean 2023-12-10T19:00:26.018 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:00:26.018 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:00:26.018 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:00:26.018 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap9-clean 2023-12-10T19:00:26.018 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9-clean --debug-rbd=20 --debug-journaler=20 2023-12-10T19:00:26.145 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stress_write_image cluster2 mirror test 2023-12-10T19:00:26.146 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:00:26.146 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:00:26.146 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:00:26.146 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-10T19:00:26.147 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local duration=8 2023-12-10T19:00:26.147 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T19:00:26.147 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-10T19:00:26.438 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:100 IOs, 38946816 bytes 2023-12-10T19:00:26.669 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:200 IOs, 97989632 bytes 2023-12-10T19:00:26.825 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:300 IOs, 126883840 bytes 2023-12-10T19:00:26.946 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:400 IOs, 154238976 bytes 2023-12-10T19:00:27.059 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:500 IOs, 165222400 bytes 2023-12-10T19:00:27.184 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:600 IOs, 189346816 bytes 2023-12-10T19:00:27.325 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:700 IOs, 219261952 bytes 2023-12-10T19:00:27.553 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:800 IOs, 287082496 bytes 2023-12-10T19:00:27.784 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:900 IOs, 380728320 bytes 2023-12-10T19:00:27.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1000 IOs, 423354368 bytes 2023-12-10T19:00:28.328 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1100 IOs, 549494784 bytes 2023-12-10T19:00:28.542 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1200 IOs, 615957504 bytes 2023-12-10T19:00:28.684 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1300 IOs, 660921344 bytes 2023-12-10T19:00:28.884 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1400 IOs, 708219904 bytes 2023-12-10T19:00:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1500 IOs, 717723648 bytes 2023-12-10T19:00:29.107 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1600 IOs, 735811584 bytes 2023-12-10T19:00:29.276 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1700 IOs, 777791488 bytes 2023-12-10T19:00:29.411 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1800 IOs, 801580032 bytes 2023-12-10T19:00:29.552 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1900 IOs, 829210624 bytes 2023-12-10T19:00:29.763 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2000 IOs, 889315328 bytes 2023-12-10T19:00:29.969 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2100 IOs, 945112064 bytes 2023-12-10T19:00:30.180 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2200 IOs, 1024411648 bytes 2023-12-10T19:00:30.439 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2300 IOs, 1091928064 bytes 2023-12-10T19:00:30.537 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2400 IOs, 1098650624 bytes 2023-12-10T19:00:30.699 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2500 IOs, 1139203072 bytes 2023-12-10T19:00:30.869 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2600 IOs, 1180123136 bytes 2023-12-10T19:00:31.088 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2700 IOs, 1243531264 bytes 2023-12-10T19:00:31.366 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2800 IOs, 1332822016 bytes 2023-12-10T19:00:31.533 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2900 IOs, 1360670720 bytes 2023-12-10T19:00:31.719 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3000 IOs, 1415868416 bytes 2023-12-10T19:00:31.945 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3100 IOs, 1482868736 bytes 2023-12-10T19:00:32.069 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3200 IOs, 1505538048 bytes 2023-12-10T19:00:32.196 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3300 IOs, 1539893248 bytes 2023-12-10T19:00:32.473 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3400 IOs, 1621956608 bytes 2023-12-10T19:00:32.633 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3500 IOs, 1660883968 bytes 2023-12-10T19:00:32.711 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3600 IOs, 1661922304 bytes 2023-12-10T19:00:32.881 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3700 IOs, 1704546304 bytes 2023-12-10T19:00:33.039 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3800 IOs, 1742230528 bytes 2023-12-10T19:00:33.371 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3900 IOs, 1856518144 bytes 2023-12-10T19:00:33.565 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4000 IOs, 1910206464 bytes 2023-12-10T19:00:33.886 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4100 IOs, 2029864960 bytes 2023-12-10T19:00:34.106 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4200 IOs, 2123332608 bytes 2023-12-10T19:00:34.152 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=124 2023-12-10T19:00:34.152 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T19:00:34.152 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 124 -eq 124 ] 2023-12-10T19:00:34.152 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:00:34.152 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-10T19:00:34.152 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:00:34.152 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:00:34.152 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:00:34.152 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state_pattern=up+replaying 2023-12-10T19:00:34.152 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local description_pattern= 2023-12-10T19:00:34.152 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local service_pattern= 2023-12-10T19:00:34.152 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:00:35.153 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-10T19:00:35.153 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:00:35.153 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:00:35.153 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:00:35.153 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state_pattern=up+replaying 2023-12-10T19:00:35.153 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local description_pattern= 2023-12-10T19:00:35.153 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local service_pattern= 2023-12-10T19:00:35.153 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-test.mirror_status 2023-12-10T19:00:35.154 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-test.mirror_status 2023-12-10T19:00:35.154 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:00:35.155 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T19:00:35.155 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2023-12-10T19:00:35.156 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ tee /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T19:00:35.244 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:test: 2023-12-10T19:00:35.244 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: global_id: c93b253b-04a0-4ada-b252-fcc724b6756e 2023-12-10T19:00:35.244 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: state: up+replaying 2023-12-10T19:00:35.245 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":498033152.0,"last_snapshot_bytes":487873536,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1702234764,"remote_snapshot_timestamp":1702234764,"replay_state":"idle"} 2023-12-10T19:00:35.245 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: service: mirror.2 on smithi076 2023-12-10T19:00:35.245 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: last_update: 2023-12-10 19:00:14 2023-12-10T19:00:35.245 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: peer_sites: 2023-12-10T19:00:35.245 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: name: cluster2 2023-12-10T19:00:35.245 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: state: up+stopped 2023-12-10T19:00:35.245 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: description: local image is primary 2023-12-10T19:00:35.245 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: last_update: 2023-12-10 19:00:18 2023-12-10T19:00:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T19:00:35.254 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: state: up+replaying 2023-12-10T19:00:35.255 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep ^ description: .* /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T19:00:35.256 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":498033152.0,"last_snapshot_bytes":487873536,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1702234764,"remote_snapshot_timestamp":1702234764,"replay_state":"idle"} 2023-12-10T19:00:35.256 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T19:00:35.257 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo up+replaying 2023-12-10T19:00:35.257 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep ^up+ 2023-12-10T19:00:35.257 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:up+replaying 2023-12-10T19:00:35.257 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep service: *mirror..* on /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T19:00:35.258 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: service: mirror.2 on smithi076 2023-12-10T19:00:35.258 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.mirror_status 2023-12-10T19:00:35.260 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local last_update=2023-12-10 19:00:14 2023-12-10T19:00:35.260 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying 2023-12-10 19:00:14 2023-12-10T19:00:35.260 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:00:35.260 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:00:35.260 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:00:35.260 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state_pattern=up+replaying 2023-12-10T19:00:35.260 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local prev_last_update=2023-12-10 19:00:14 2023-12-10T19:00:35.261 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror.mirror_status 2023-12-10T19:00:35.261 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror.mirror_status 2023-12-10T19:00:35.261 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:00:35.261 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror.mirror_status 2023-12-10T19:00:35.261 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-10T19:00:35.315 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: failed to query services: (13) Permission denied 2023-12-10T19:00:35.372 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local last_update state 2023-12-10T19:00:35.372 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2023-12-10T19:00:35.375 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ last_update=2023-12-10 19:00:14 2023-12-10T19:00:35.375 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2023-12-10T19:00:35.379 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ state=up+replaying 2023-12-10T19:00:35.379 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo up+replaying 2023-12-10T19:00:35.379 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep up+replaying 2023-12-10T19:00:35.380 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:up+replaying 2023-12-10T19:00:35.380 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:00:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:00:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ snap_name=snap10 2023-12-10T19:00:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror test snap10 2023-12-10T19:00:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:00:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:00:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:00:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap10 2023-12-10T19:00:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10 --debug-rbd=20 --debug-journaler=20 2023-12-10T19:01:02.365 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-10T19:01:02.365 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:01:02.365 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:01:02.365 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:01:02.365 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-10T19:01:02.366 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:01:02.366 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:01:02.366 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:01:02.366 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=started 2023-12-10T19:01:02.366 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:01:02.366 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:01:03.365 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-10T19:01:03.365 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:01:03.365 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:01:03.365 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:01:03.365 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=started 2023-12-10T19:01:03.365 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_result 2023-12-10T19:01:03.365 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=stopped 2023-12-10T19:01:03.366 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemons cluster1 rbd+ mirror status mirror/test 2023-12-10T19:01:03.366 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_instance=cluster1grep 2023-12-10T19:01:03.366 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: -i state+ 2023-12-10T19:01:03.366 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:shift 2023-12-10T19:01:03.366 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:01:03.366 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance=cluster1 2023-12-10T19:01:03.366 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loop_instance 2023-12-10T19:01:03.367 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:01:03.367 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T19:01:03.367 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T19:01:03.368 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-10T19:01:03.368 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T19:01:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:01:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:01:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T19:01:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:01:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:01:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:01:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:01:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:01:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:01:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:01:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:01:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:01:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:01:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T19:01:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T19:01:03.370 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:01:03.370 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:01:03.370 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T19:01:03.370 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:01:03.370 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:01:03.370 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:01:03.370 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:01:03.370 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:01:03.370 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:01:03.370 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:01:03.370 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:01:03.370 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:01:03.370 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:01:03.370 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T19:01:03.394 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:01:03.394 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:01:03.394 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:01:03.394 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-10T19:01:03.499 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-10T19:01:03.499 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:01:03.499 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:01:03.502 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-10T19:01:03.502 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:1 2023-12-10T19:01:03.502 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:01:03.502 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:01:03.502 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-10T19:01:03.502 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:01:03.502 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:01:03.502 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:01:03.503 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:01:03.503 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:01:03.503 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:01:03.503 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:01:03.503 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:01:03.503 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:01:03.503 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:01:03.503 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-10T19:01:03.503 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:1 2023-12-10T19:01:03.503 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:01:03.503 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:01:03.503 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-10T19:01:03.503 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:01:03.503 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:01:03.503 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:01:03.503 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:01:03.504 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:01:03.504 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:01:03.504 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:01:03.504 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:01:03.504 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:01:03.504 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:01:03.504 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-10T19:01:03.526 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:01:03.526 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:01:03.526 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:01:03.526 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-10T19:01:03.640 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-10T19:01:03.641 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:01:03.641 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:01:03.643 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-10T19:01:03.643 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:2 2023-12-10T19:01:03.643 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:01:03.643 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:01:03.643 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-10T19:01:03.643 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:01:03.643 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:01:03.643 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:01:03.643 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:01:03.643 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:01:03.644 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:01:03.644 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:01:03.644 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:01:03.644 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:01:03.644 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:01:03.644 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-10T19:01:03.644 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:2 2023-12-10T19:01:03.644 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:01:03.644 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:01:03.644 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-10T19:01:03.644 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:01:03.644 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:01:03.644 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:01:03.644 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:01:03.644 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:01:03.645 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:01:03.645 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:01:03.645 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:01:03.645 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:01:03.645 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:01:03.645 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-10T19:01:03.667 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:01:03.667 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:01:03.667 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:01:03.668 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-10T19:01:03.793 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:01:03.793 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ status_result= "state": "Replaying" 2023-12-10T19:01:03.793 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ + grep -i Replaying 2023-12-10T19:01:03.793 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:echo "state": "Replaying" 2023-12-10T19:01:03.794 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "state": "Replaying" 2023-12-10T19:01:03.795 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=started 2023-12-10T19:01:03.795 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test started = started 2023-12-10T19:01:03.795 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:01:03.795 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-10T19:01:03.796 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:01:03.796 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:01:03.796 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:01:03.796 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:01:03.796 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = journal ] 2023-12-10T19:01:03.796 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = snapshot ] 2023-12-10T19:01:03.796 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-10T19:01:03.796 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:01:03.796 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:01:03.796 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:01:03.796 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:01:03.796 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster2-mirror-test.status 2023-12-10T19:01:03.796 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2-mirror-test.status 2023-12-10T19:01:03.796 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:01:03.797 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-test.status 2023-12-10T19:01:03.797 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-test.status 2023-12-10T19:01:03.797 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-test.status 2023-12-10T19:01:03.797 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:01:03.798 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T19:01:03.798 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-10T19:01:03.798 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:01:03.798 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:01:03.798 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:01:03.798 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-10T19:01:05.818 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Snapshot ID: 41 2023-12-10T19:01:05.835 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.status 2023-12-10T19:01:05.835 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:01:05.835 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:01:05.836 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:01:05.836 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-test.status 2023-12-10T19:01:05.836 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:01:05.839 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-10T19:01:05.916 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-10T19:01:05.918 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snapshot_id=41 2023-12-10T19:01:05.919 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:01:05.919 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.2 2023-12-10T19:01:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T19:01:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:01:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:01:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:01:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T19:01:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-10T19:01:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:01:06.209 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:01:06.209 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:01:06.211 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:01:06.211 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:01:06.211 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:01:06.212 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:01:06.212 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 41 = 2023-12-10T19:01:06.212 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.4 2023-12-10T19:01:06.613 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T19:01:06.614 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:01:06.614 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:01:06.614 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:01:06.614 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T19:01:06.614 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-10T19:01:06.614 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:01:06.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:01:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:01:06.706 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:01:06.706 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:01:06.706 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:01:06.706 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:01:06.707 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 41 = 2023-12-10T19:01:06.707 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.8 2023-12-10T19:01:07.508 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T19:01:07.508 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:01:07.508 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:01:07.508 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:01:07.509 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T19:01:07.509 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-10T19:01:07.509 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:01:07.597 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:01:07.598 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:01:07.601 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:01:07.601 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:01:07.601 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:01:07.601 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:01:07.601 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 41 = 2023-12-10T19:01:07.601 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1.6 2023-12-10T19:01:08.715 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.8703995298846078) > weight (0.25) 2023-12-10T19:01:08.715 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.9703158687494418) > weight (0.25) 2023-12-10T19:01:08.716 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2023-12-10T19:01:08.716 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2023-12-10T19:01:08.716 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2023-12-10T19:01:08.717 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi076.stderr:2023-12-10T19:01:08.716+0000 7f6bc8343640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 42491) UID: 1000 2023-12-10T19:01:09.202 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T19:01:09.202 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:01:09.202 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:01:09.202 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:01:09.202 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T19:01:09.202 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-10T19:01:09.202 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:01:09.298 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:01:09.301 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=41 2023-12-10T19:01:09.301 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 41 = 41 2023-12-10T19:01:09.301 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:01:09.301 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snap_present cluster1 mirror test snap10 2023-12-10T19:01:09.301 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:01:09.301 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:01:09.301 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:01:09.301 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap10 2023-12-10T19:01:09.301 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:01:09.301 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:01:10.302 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 info mirror/test@snap10 2023-12-10T19:01:10.387 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'test': 2023-12-10T19:01:10.387 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 512 MiB in 128 objects 2023-12-10T19:01:10.387 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:01:10.387 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 20 2023-12-10T19:01:10.388 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 10c0be4f5681 2023-12-10T19:01:10.388 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.10c0be4f5681 2023-12-10T19:01:10.388 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:01:10.388 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering, non-primary 2023-12-10T19:01:10.388 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:01:10.388 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:01:10.388 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 18:52:46 2023 2023-12-10T19:01:10.388 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 18:59:31 2023 2023-12-10T19:01:10.388 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 18:52:46 2023 2023-12-10T19:01:10.389 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: protected: False 2023-12-10T19:01:10.389 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:01:10.389 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:01:10.389 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: c93b253b-04a0-4ada-b252-fcc724b6756e 2023-12-10T19:01:10.389 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: false 2023-12-10T19:01:10.395 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:01:10.395 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n snap9-clean ] 2023-12-10T19:01:10.395 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror test snap9-clean 2023-12-10T19:01:10.395 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:01:10.395 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:01:10.395 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap9-clean 2023-12-10T19:01:10.395 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:01:10.395 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T19:01:10.395 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:10.395 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:10.396 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T19:01:11.099 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T19:01:11.110 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:12.214 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T19:01:12.239 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:12.517 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:12.652 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:01:12.653 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror test snap10 2023-12-10T19:01:12.653 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:01:12.653 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:01:12.653 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap10 2023-12-10T19:01:12.653 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:01:12.653 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T19:01:12.653 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:12.653 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:12.654 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T19:01:13.323 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T19:01:13.338 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:14.469 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T19:01:14.491 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:14.771 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:14.907 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:01:14.907 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ clean_snap_name=snap10-clean 2023-12-10T19:01:14.907 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror test snap10-clean 2023-12-10T19:01:14.907 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:01:14.907 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:01:14.907 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:01:14.907 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap10-clean 2023-12-10T19:01:14.907 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10-clean --debug-rbd=20 --debug-journaler=20 2023-12-10T19:01:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-10T19:01:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:01:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:01:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:01:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-10T19:01:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:01:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:01:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:01:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=started 2023-12-10T19:01:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:01:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:01:16.717 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2023-12-10T19:01:16.717 INFO:teuthology.orchestra.run:waiting for 600 2023-12-10T19:01:16.718 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2023-12-10T19:01:16.718 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2023-12-10T19:01:16.718 DEBUG:teuthology.orchestra.run.smithi076:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2023-12-10T19:01:16.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2023-12-10T19:01:16.721 INFO:tasks.rbd_mirror_thrash:waiting for 66 secs before thrashing 2023-12-10T19:01:16.803 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-10T19:01:16.803 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:01:16.803 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:01:16.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:01:16.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=started 2023-12-10T19:01:16.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_result 2023-12-10T19:01:16.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=stopped 2023-12-10T19:01:16.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-10T19:01:16.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_instance=cluster1 2023-12-10T19:01:16.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ + shift 2023-12-10T19:01:16.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:01:16.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance=cluster1 2023-12-10T19:01:16.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ greplocal -i loop_instance 2023-12-10T19:01:16.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: state 2023-12-10T19:01:16.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:01:16.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T19:01:16.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T19:01:16.806 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-10T19:01:16.806 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T19:01:16.806 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:01:16.806 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:01:16.806 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T19:01:16.806 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:01:16.806 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:01:16.807 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:01:16.807 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:01:16.807 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:01:16.807 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:01:16.807 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:01:16.807 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:01:16.807 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:01:16.807 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:01:16.807 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T19:01:16.807 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T19:01:16.807 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:01:16.807 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:01:16.807 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T19:01:16.807 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:01:16.808 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:01:16.808 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:01:16.808 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:01:16.808 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:01:16.808 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:01:16.808 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:01:16.808 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:01:16.809 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:01:16.809 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:01:16.809 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T19:01:16.826 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:01:16.826 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:01:16.826 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:01:16.826 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-10T19:01:16.939 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-10T19:01:16.939 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:01:16.939 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:01:16.942 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-10T19:01:16.942 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:1 2023-12-10T19:01:16.942 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:01:16.942 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:01:16.942 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-10T19:01:16.942 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:01:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:01:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:01:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:01:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:01:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:01:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:01:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:01:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:01:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:01:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-10T19:01:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:1 2023-12-10T19:01:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:01:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:01:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-10T19:01:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:01:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:01:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:01:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:01:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:01:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:01:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:01:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:01:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:01:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:01:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-10T19:01:16.963 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:01:16.963 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:01:16.963 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:01:16.963 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-10T19:01:17.063 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-10T19:01:17.064 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:01:17.064 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:01:17.065 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-10T19:01:17.066 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:2 2023-12-10T19:01:17.066 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:01:17.066 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:01:17.066 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-10T19:01:17.066 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:01:17.066 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:01:17.066 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:01:17.066 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:01:17.066 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:01:17.066 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:01:17.066 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:01:17.066 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:01:17.066 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:01:17.066 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:01:17.066 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-10T19:01:17.067 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:2 2023-12-10T19:01:17.067 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:01:17.067 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:01:17.067 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-10T19:01:17.067 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:01:17.067 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:01:17.067 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:01:17.067 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:01:17.067 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:01:17.067 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:01:17.067 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:01:17.067 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:01:17.067 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:01:17.067 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:01:17.067 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-10T19:01:17.089 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:01:17.089 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:01:17.089 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:01:17.090 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-10T19:01:17.201 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:01:17.202 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ status_result= "state": "Replaying" 2023-12-10T19:01:17.202 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo "state": "Replaying" 2023-12-10T19:01:17.202 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep -i Replaying 2023-12-10T19:01:17.204 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "state": "Replaying" 2023-12-10T19:01:17.205 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=started 2023-12-10T19:01:17.205 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test started = started 2023-12-10T19:01:17.205 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:01:17.205 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-10T19:01:17.205 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:01:17.205 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:01:17.205 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:01:17.206 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:01:17.206 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = journal ] 2023-12-10T19:01:17.206 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = snapshot ] 2023-12-10T19:01:17.206 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-10T19:01:17.206 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:01:17.206 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:01:17.206 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:01:17.206 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:01:17.206 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster2-mirror-test.status 2023-12-10T19:01:17.206 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2-mirror-test.status 2023-12-10T19:01:17.206 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:01:17.212 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-test.status 2023-12-10T19:01:17.212 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-test.status 2023-12-10T19:01:17.212 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-test.status 2023-12-10T19:01:17.212 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:01:17.212 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T19:01:17.212 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-10T19:01:17.212 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:01:17.212 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:01:17.212 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:01:17.212 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-10T19:01:18.843 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Snapshot ID: 44 2023-12-10T19:01:18.865 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.status 2023-12-10T19:01:18.865 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:01:18.865 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:01:18.865 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:01:18.865 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-test.status 2023-12-10T19:01:18.865 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-10T19:01:18.865 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:01:18.964 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-10T19:01:18.967 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snapshot_id=44 2023-12-10T19:01:18.967 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:01:18.967 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.2 2023-12-10T19:01:19.169 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T19:01:19.169 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:01:19.169 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:01:19.169 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:01:19.169 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.status 2023-12-10T19:01:19.169 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-10T19:01:19.169 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:01:19.263 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:01:19.266 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=44 2023-12-10T19:01:19.266 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 44 = 44 2023-12-10T19:01:19.266 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:01:19.266 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snap_present cluster1 mirror test snap10-clean 2023-12-10T19:01:19.266 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:01:19.266 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:01:19.266 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:01:19.266 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap10-clean 2023-12-10T19:01:19.266 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:01:19.266 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:01:20.267 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 info mirror/test@snap10-clean 2023-12-10T19:01:20.340 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'test': 2023-12-10T19:01:20.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 512 MiB in 128 objects 2023-12-10T19:01:20.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:01:20.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 21 2023-12-10T19:01:20.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 10c0be4f5681 2023-12-10T19:01:20.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.10c0be4f5681 2023-12-10T19:01:20.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:01:20.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering, non-primary 2023-12-10T19:01:20.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:01:20.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:01:20.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 18:52:46 2023 2023-12-10T19:01:20.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:01:11 2023 2023-12-10T19:01:20.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 18:52:46 2023 2023-12-10T19:01:20.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: protected: False 2023-12-10T19:01:20.342 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:01:20.342 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:01:20.342 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: c93b253b-04a0-4ada-b252-fcc724b6756e 2023-12-10T19:01:20.342 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: false 2023-12-10T19:01:20.346 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:01:20.346 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 1 10 2023-12-10T19:01:20.347 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ snap_name=snap1 2023-12-10T19:01:20.347 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror test snap1 2023-12-10T19:01:20.347 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:01:20.347 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:01:20.347 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap1 2023-12-10T19:01:20.347 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:01:20.347 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T19:01:20.347 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:20.347 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T19:01:21.102 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T19:01:21.123 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:22.272 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T19:01:22.294 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:22.577 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:22.712 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:01:22.712 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ snap_name=snap1-clean 2023-12-10T19:01:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror test snap1-clean 2023-12-10T19:01:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:01:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:01:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap1-clean 2023-12-10T19:01:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:01:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T19:01:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T19:01:23.366 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T19:01:23.378 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:24.544 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T19:01:24.570 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:24.852 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:24.988 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:01:24.988 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ snap_name=snap2 2023-12-10T19:01:24.988 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror test snap2 2023-12-10T19:01:24.988 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:01:24.988 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:01:24.988 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap2 2023-12-10T19:01:24.989 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:01:24.989 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T19:01:24.989 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:24.989 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:24.989 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T19:01:25.696 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T19:01:25.713 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:26.954 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T19:01:26.974 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:27.254 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:27.391 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:01:27.391 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ snap_name=snap2-clean 2023-12-10T19:01:27.391 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror test snap2-clean 2023-12-10T19:01:27.391 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:01:27.391 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:01:27.392 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap2-clean 2023-12-10T19:01:27.392 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:01:27.392 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T19:01:27.392 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:27.392 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:27.393 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T19:01:28.092 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T19:01:28.112 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:29.306 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T19:01:29.327 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:29.606 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:29.742 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:01:29.742 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ snap_name=snap3 2023-12-10T19:01:29.742 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror test snap3 2023-12-10T19:01:29.742 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:01:29.742 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:01:29.742 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap3 2023-12-10T19:01:29.742 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:01:29.743 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T19:01:29.743 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:29.743 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:29.743 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T19:01:30.475 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T19:01:30.493 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:31.533 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T19:01:31.554 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:31.835 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:31.970 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:01:31.970 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ snap_name=snap3-clean 2023-12-10T19:01:31.970 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror test snap3-clean 2023-12-10T19:01:31.971 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:01:31.971 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:01:31.971 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap3-clean 2023-12-10T19:01:31.971 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:01:31.971 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T19:01:31.971 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:31.971 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:31.971 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T19:01:32.670 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T19:01:32.743 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:33.820 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T19:01:33.837 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:34.116 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:34.254 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:01:34.254 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ snap_name=snap4 2023-12-10T19:01:34.254 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror test snap4 2023-12-10T19:01:34.254 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:01:34.254 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:01:34.255 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap4 2023-12-10T19:01:34.255 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:01:34.255 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T19:01:34.255 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:34.255 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:34.255 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T19:01:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T19:01:34.965 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:36.029 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T19:01:36.048 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:36.327 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:01:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ snap_name=snap4-clean 2023-12-10T19:01:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror test snap4-clean 2023-12-10T19:01:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:01:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:01:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap4-clean 2023-12-10T19:01:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:01:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T19:01:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:36.464 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T19:01:37.174 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T19:01:37.192 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:37.727 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.4177957449424917) > weight (0.25) 2023-12-10T19:01:37.727 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.8776104675358897) > weight (0.25) 2023-12-10T19:01:37.727 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.7078857168084802) > weight (0.25) 2023-12-10T19:01:37.728 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.951344216942375) > weight (0.25) 2023-12-10T19:01:37.728 INFO:tasks.rbd_mirror_thrash:waiting for 63 secs before thrashing 2023-12-10T19:01:38.322 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T19:01:38.345 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:38.625 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:38.761 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:01:38.761 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ snap_name=snap5 2023-12-10T19:01:38.761 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror test snap5 2023-12-10T19:01:38.761 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:01:38.761 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:01:38.761 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap5 2023-12-10T19:01:38.762 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:01:38.762 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T19:01:38.762 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:38.762 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:38.762 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T19:01:39.452 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T19:01:39.465 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:40.661 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T19:01:40.683 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:40.964 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:41.103 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:01:41.103 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ snap_name=snap5-clean 2023-12-10T19:01:41.103 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror test snap5-clean 2023-12-10T19:01:41.103 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:01:41.104 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:01:41.104 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap5-clean 2023-12-10T19:01:41.104 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:01:41.104 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T19:01:41.104 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:41.104 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:41.104 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T19:01:41.825 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T19:01:41.838 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:42.984 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T19:01:43.004 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:43.284 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:43.422 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:01:43.422 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ snap_name=snap6 2023-12-10T19:01:43.422 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror test snap6 2023-12-10T19:01:43.422 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:01:43.422 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:01:43.422 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap6 2023-12-10T19:01:43.423 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:01:43.423 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T19:01:43.423 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:43.423 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:43.423 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T19:01:44.096 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T19:01:44.109 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:45.181 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T19:01:45.202 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:45.483 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:45.620 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:01:45.621 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ snap_name=snap6-clean 2023-12-10T19:01:45.621 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror test snap6-clean 2023-12-10T19:01:45.621 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:01:45.621 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:01:45.621 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap6-clean 2023-12-10T19:01:45.621 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:01:45.621 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T19:01:45.621 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:45.621 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:45.621 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T19:01:46.265 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T19:01:46.279 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:47.616 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T19:01:47.638 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:47.918 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:48.054 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:01:48.055 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ snap_name=snap7 2023-12-10T19:01:48.055 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror test snap7 2023-12-10T19:01:48.055 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:01:48.055 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:01:48.055 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap7 2023-12-10T19:01:48.055 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:01:48.055 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T19:01:48.055 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:48.055 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:48.055 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T19:01:48.751 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T19:01:48.764 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:49.952 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T19:01:49.972 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:50.266 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:50.404 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:01:50.404 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ snap_name=snap7-clean 2023-12-10T19:01:50.404 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror test snap7-clean 2023-12-10T19:01:50.404 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:01:50.404 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:01:50.404 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap7-clean 2023-12-10T19:01:50.404 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:01:50.404 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T19:01:50.405 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:50.405 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:50.405 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T19:01:51.106 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T19:01:51.121 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:52.192 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T19:01:52.208 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:52.489 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:01:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ snap_name=snap8 2023-12-10T19:01:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror test snap8 2023-12-10T19:01:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:01:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:01:52.628 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap8 2023-12-10T19:01:52.628 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:01:52.628 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T19:01:52.628 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:52.628 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:52.628 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T19:01:53.382 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T19:01:53.400 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:54.597 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T19:01:54.621 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:54.904 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:55.042 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:01:55.042 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ snap_name=snap8-clean 2023-12-10T19:01:55.042 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror test snap8-clean 2023-12-10T19:01:55.042 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:01:55.042 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:01:55.042 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap8-clean 2023-12-10T19:01:55.042 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:01:55.042 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T19:01:55.042 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:55.042 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:55.043 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T19:01:55.740 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T19:01:55.759 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:56.997 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T19:01:57.012 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:57.290 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:57.427 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:01:57.428 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ snap_name=snap9 2023-12-10T19:01:57.428 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror test snap9 2023-12-10T19:01:57.428 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:01:57.428 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:01:57.428 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap9 2023-12-10T19:01:57.428 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:01:57.428 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T19:01:57.428 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:57.428 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:57.428 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T19:01:58.061 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T19:01:58.073 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:59.380 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T19:01:59.407 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:59.688 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:59.826 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:01:59.826 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ snap_name=snap9-clean 2023-12-10T19:01:59.826 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror test snap9-clean 2023-12-10T19:01:59.826 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:01:59.826 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:01:59.827 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap9-clean 2023-12-10T19:01:59.827 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:01:59.827 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T19:01:59.827 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:59.827 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:01:59.827 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T19:02:00.517 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T19:02:00.533 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:02:01.848 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T19:02:01.867 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:02:02.149 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:02:02.287 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:02:02.287 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ snap_name=snap10 2023-12-10T19:02:02.287 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror test snap10 2023-12-10T19:02:02.287 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:02:02.288 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:02:02.288 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap10 2023-12-10T19:02:02.288 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:02:02.288 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T19:02:02.288 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:02:02.288 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:02:02.288 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T19:02:03.001 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T19:02:03.015 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:02:04.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T19:02:04.160 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:02:04.441 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:02:04.581 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:02:04.581 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ snap_name=snap10-clean 2023-12-10T19:02:04.581 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror test snap10-clean 2023-12-10T19:02:04.581 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:02:04.581 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:02:04.581 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap10-clean 2023-12-10T19:02:04.581 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:02:04.581 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T19:02:04.581 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:02:04.581 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:02:04.582 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10-clean /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export 2023-12-10T19:02:05.283 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T19:02:05.299 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10-clean /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:02:06.496 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-10T19:02:06.526 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:02:06.808 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-test.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-test.export 2023-12-10T19:02:06.946 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:02:06.946 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 1 10 2023-12-10T19:02:06.947 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ snap_name=snap1 2023-12-10T19:02:06.947 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror test snap1 2023-12-10T19:02:06.947 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:02:06.947 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:02:06.947 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:02:06.947 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap1 2023-12-10T19:02:06.948 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1 2023-12-10T19:02:07.375 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:02:07.398 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ snap_name=snap1-clean 2023-12-10T19:02:07.398 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror test snap1-clean 2023-12-10T19:02:07.398 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:02:07.398 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:02:07.398 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:02:07.399 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap1-clean 2023-12-10T19:02:07.399 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1-clean 2023-12-10T19:02:08.404 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:02:08.419 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ snap_name=snap2 2023-12-10T19:02:08.419 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror test snap2 2023-12-10T19:02:08.419 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:02:08.420 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:02:08.420 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:02:08.420 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap2 2023-12-10T19:02:08.420 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2 2023-12-10T19:02:09.397 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:02:09.411 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ snap_name=snap2-clean 2023-12-10T19:02:09.412 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror test snap2-clean 2023-12-10T19:02:09.412 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:02:09.412 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:02:09.412 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:02:09.412 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap2-clean 2023-12-10T19:02:09.412 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2-clean 2023-12-10T19:02:10.408 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:02:10.431 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ snap_name=snap3 2023-12-10T19:02:10.431 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror test snap3 2023-12-10T19:02:10.431 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:02:10.431 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:02:10.431 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:02:10.431 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap3 2023-12-10T19:02:10.431 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3 2023-12-10T19:02:11.413 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:02:11.429 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ snap_name=snap3-clean 2023-12-10T19:02:11.429 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror test snap3-clean 2023-12-10T19:02:11.429 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:02:11.429 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:02:11.430 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:02:11.430 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap3-clean 2023-12-10T19:02:11.430 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3-clean 2023-12-10T19:02:12.441 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:02:12.454 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ snap_name=snap4 2023-12-10T19:02:12.454 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror test snap4 2023-12-10T19:02:12.454 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:02:12.454 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:02:12.454 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:02:12.454 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap4 2023-12-10T19:02:12.454 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4 2023-12-10T19:02:13.436 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:02:13.449 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ snap_name=snap4-clean 2023-12-10T19:02:13.449 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror test snap4-clean 2023-12-10T19:02:13.449 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:02:13.449 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:02:13.449 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:02:13.449 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap4-clean 2023-12-10T19:02:13.449 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4-clean 2023-12-10T19:02:13.847 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:02:13.861 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ snap_name=snap5 2023-12-10T19:02:13.861 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror test snap5 2023-12-10T19:02:13.861 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:02:13.862 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:02:13.862 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:02:13.862 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap5 2023-12-10T19:02:13.862 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5 2023-12-10T19:02:14.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:02:14.833 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ snap_name=snap5-clean 2023-12-10T19:02:14.834 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror test snap5-clean 2023-12-10T19:02:14.834 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:02:14.834 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:02:14.834 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:02:14.834 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap5-clean 2023-12-10T19:02:14.834 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5-clean 2023-12-10T19:02:15.846 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:02:15.875 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ snap_name=snap6 2023-12-10T19:02:15.875 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror test snap6 2023-12-10T19:02:15.875 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:02:15.875 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:02:15.875 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:02:15.875 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap6 2023-12-10T19:02:15.875 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6 2023-12-10T19:02:16.822 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:02:16.844 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ snap_name=snap6-clean 2023-12-10T19:02:16.844 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror test snap6-clean 2023-12-10T19:02:16.844 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:02:16.844 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:02:16.845 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:02:16.845 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap6-clean 2023-12-10T19:02:16.845 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6-clean 2023-12-10T19:02:17.848 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:02:17.870 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ snap_name=snap7 2023-12-10T19:02:17.870 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror test snap7 2023-12-10T19:02:17.870 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:02:17.871 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:02:17.871 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:02:17.871 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap7 2023-12-10T19:02:17.871 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7 2023-12-10T19:02:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:02:18.844 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ snap_name=snap7-clean 2023-12-10T19:02:18.844 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror test snap7-clean 2023-12-10T19:02:18.844 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:02:18.844 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:02:18.844 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:02:18.844 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap7-clean 2023-12-10T19:02:18.844 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7-clean 2023-12-10T19:02:19.865 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:02:19.869 INFO:tasks.ceph.cluster2.osd.2.smithi076.stderr:2023-12-10T19:02:19.867+0000 7f7f6add3640 -1 reset not still connected to 0x564305c73040 2023-12-10T19:02:19.897 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ snap_name=snap8 2023-12-10T19:02:19.897 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror test snap8 2023-12-10T19:02:19.897 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:02:19.897 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:02:19.897 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:02:19.897 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap8 2023-12-10T19:02:19.897 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8 2023-12-10T19:02:20.866 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:02:20.886 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ snap_name=snap8-clean 2023-12-10T19:02:20.886 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror test snap8-clean 2023-12-10T19:02:20.886 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:02:20.886 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:02:20.886 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:02:20.886 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap8-clean 2023-12-10T19:02:20.886 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8-clean 2023-12-10T19:02:21.893 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:02:21.914 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ snap_name=snap9 2023-12-10T19:02:21.914 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror test snap9 2023-12-10T19:02:21.914 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:02:21.914 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:02:21.914 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:02:21.914 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap9 2023-12-10T19:02:21.914 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9 2023-12-10T19:02:22.722 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.3469725135439753) > weight (0.25) 2023-12-10T19:02:22.722 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2023-12-10T19:02:22.722 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2023-12-10T19:02:22.722 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2023-12-10T19:02:22.723 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr:2023-12-10T19:02:22.723+0000 7f7830416640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 41201) UID: 1000 2023-12-10T19:02:22.899 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:02:22.913 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ snap_name=snap9-clean 2023-12-10T19:02:22.913 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror test snap9-clean 2023-12-10T19:02:22.913 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:02:22.913 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:02:22.914 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:02:22.914 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap9-clean 2023-12-10T19:02:22.914 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9-clean 2023-12-10T19:02:23.846 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:02:23.865 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ snap_name=snap10 2023-12-10T19:02:23.865 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror test snap10 2023-12-10T19:02:23.866 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:02:23.866 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:02:23.866 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:02:23.866 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap10 2023-12-10T19:02:23.866 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10 2023-12-10T19:02:24.892 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:02:24.906 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ snap_name=snap10-clean 2023-12-10T19:02:24.906 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror test snap10-clean 2023-12-10T19:02:24.906 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:02:24.906 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:02:24.906 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:02:24.906 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap10-clean 2023-12-10T19:02:24.906 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10-clean 2023-12-10T19:02:25.906 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:02:25.918 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image_retry cluster2 mirror test 2023-12-10T19:02:25.918 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:02:25.918 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:02:25.918 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:02:25.918 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:02:25.919 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror test 2023-12-10T19:02:25.919 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:02:25.919 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:02:25.919 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:02:25.919 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/test 2023-12-10T19:02:25.987 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/test 2023-12-10T19:02:26.143 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi076.stderr:2023-12-10T19:02:26.143+0000 7f34cc858640 -1 rbd::mirror::ImageMap: 0x562ef9c5e900 handle_peer_ack_remove: failed to remove global_image_id=c93b253b-04a0-4ada-b252-fcc724b6756e 2023-12-10T19:02:27.335 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-10T19:02:27.339 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:02:27.339 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_present cluster1 mirror test deleted 2023-12-10T19:02:27.339 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:02:27.339 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:02:27.339 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:02:27.339 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=deleted 2023-12-10T19:02:27.339 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:02:27.340 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:02:27.340 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:02:27.340 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror test 2023-12-10T19:02:27.340 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:02:27.340 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:02:27.340 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:02:27.340 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/test 2023-12-10T19:02:27.340 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:02:27.386 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image test: (2) No such file or directory 2023-12-10T19:02:27.391 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image_id= 2023-12-10T19:02:27.391 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.1 2023-12-10T19:02:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror test deleted 2023-12-10T19:02:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:02:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:02:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:02:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=deleted 2023-12-10T19:02:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:02:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:02:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:02:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror test 2023-12-10T19:02:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:02:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:02:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=test 2023-12-10T19:02:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/test 2023-12-10T19:02:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:02:27.535 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image test: (2) No such file or directory 2023-12-10T19:02:27.540 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:02:27.540 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:02:27.540 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test deleted = deleted 2023-12-10T19:02:27.541 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:02:27.541 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ testlog TEST: create many images 2023-12-10T19:02:27.541 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ tee -a /tmp/tmp.qNjRyMq75D/rbd-mirror.test.log 2023-12-10T19:02:27.541 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ date +%F %T 2023-12-10T19:02:27.541 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo 2023-12-10 19:02:27 TEST: create many images 2023-12-10T19:02:27.541 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:2023-12-10 19:02:27 TEST: create many images 2023-12-10T19:02:27.541 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ snap_name=snap 2023-12-10T19:02:27.541 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 1 50 2023-12-10T19:02:27.541 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_1 2023-12-10T19:02:27.541 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image_and_enable_mirror cluster2 mirror image_1 snapshot 128M 2023-12-10T19:02:27.541 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:02:27.541 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:02:27.541 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:02:27.541 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:02:27.542 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_1 2023-12-10T19:02:27.542 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:02:27.542 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:02:27.542 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n snapshot ] 2023-12-10T19:02:27.542 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:02:27.542 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image cluster2 mirror image_1 128M 2023-12-10T19:02:27.542 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:02:27.542 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:02:27.542 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:02:27.542 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:02:27.542 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_1 2023-12-10T19:02:27.542 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:02:27.542 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local size=128 2023-12-10T19:02:27.542 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n 128M ] 2023-12-10T19:02:27.542 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ size=128M 2023-12-10T19:02:27.542 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:02:27.543 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_1 2023-12-10T19:02:27.587 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ is_pool_mirror_mode_image mirror 2023-12-10T19:02:27.587 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:02:27.587 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ image = image ] 2023-12-10T19:02:27.587 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:02:27.587 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ enable_mirror cluster2 mirror image_1 snapshot 2023-12-10T19:02:27.587 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:02:27.587 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:02:27.587 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_1 2023-12-10T19:02:27.588 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:02:27.588 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 snapshot 2023-12-10T19:02:27.723 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2023-12-10T19:02:27.724 INFO:teuthology.orchestra.run:waiting for 600 2023-12-10T19:02:27.724 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2023-12-10T19:02:27.724 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-12-10T19:02:27.725 DEBUG:teuthology.orchestra.run.smithi076:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-12-10T19:02:27.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-12-10T19:02:27.729 INFO:tasks.rbd_mirror_thrash:waiting for 71 secs before thrashing 2023-12-10T19:02:27.936 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirroring enabled 2023-12-10T19:02:27.942 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2023-12-10T19:02:27.981 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_1': 2023-12-10T19:02:27.981 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:02:27.981 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:02:27.981 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 1 2023-12-10T19:02:27.981 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 1264308c46e9 2023-12-10T19:02:27.981 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.1264308c46e9 2023-12-10T19:02:27.982 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:02:27.982 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering 2023-12-10T19:02:27.982 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:02:27.982 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:02:27.982 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:02:27 2023 2023-12-10T19:02:27.982 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:02:27 2023 2023-12-10T19:02:27.982 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:02:27 2023 2023-12-10T19:02:27.982 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:02:27.982 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:02:27.982 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: 8fd23019-e0bb-4243-ad49-4fbc841c8b88 2023-12-10T19:02:27.982 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: true 2023-12-10T19:02:27.985 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T19:02:27.985 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stress_write_image cluster2 mirror image_1 2023-12-10T19:02:27.985 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:02:27.985 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:02:27.985 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_1 2023-12-10T19:02:27.985 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-10T19:02:27.986 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local duration=13 2023-12-10T19:02:27.986 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T19:02:27.986 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2023-12-10T19:02:28.141 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:100 IOs, 18163712 bytes 2023-12-10T19:02:28.293 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:200 IOs, 55091200 bytes 2023-12-10T19:02:28.457 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:300 IOs, 105225216 bytes 2023-12-10T19:02:28.554 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:400 IOs, 109201408 bytes 2023-12-10T19:02:28.705 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:500 IOs, 155965440 bytes 2023-12-10T19:02:28.830 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:600 IOs, 167510016 bytes 2023-12-10T19:02:28.960 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:700 IOs, 197970944 bytes 2023-12-10T19:02:29.086 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:800 IOs, 209822720 bytes 2023-12-10T19:02:29.221 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:900 IOs, 240676864 bytes 2023-12-10T19:02:29.327 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1000 IOs, 275232768 bytes 2023-12-10T19:02:29.560 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1100 IOs, 329421824 bytes 2023-12-10T19:02:29.753 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1200 IOs, 388787200 bytes 2023-12-10T19:02:29.861 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1300 IOs, 403959808 bytes 2023-12-10T19:02:30.046 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1400 IOs, 463405056 bytes 2023-12-10T19:02:30.188 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1500 IOs, 516975616 bytes 2023-12-10T19:02:30.304 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1600 IOs, 526723072 bytes 2023-12-10T19:02:30.409 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1700 IOs, 536872960 bytes 2023-12-10T19:02:30.733 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1800 IOs, 641311744 bytes 2023-12-10T19:02:30.868 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1900 IOs, 666595328 bytes 2023-12-10T19:02:30.975 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2000 IOs, 682635264 bytes 2023-12-10T19:02:31.279 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2100 IOs, 809027584 bytes 2023-12-10T19:02:31.520 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2200 IOs, 866089984 bytes 2023-12-10T19:02:31.711 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2300 IOs, 919482368 bytes 2023-12-10T19:02:31.857 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2400 IOs, 958186496 bytes 2023-12-10T19:02:31.940 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2500 IOs, 958786560 bytes 2023-12-10T19:02:32.157 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2600 IOs, 1019978752 bytes 2023-12-10T19:02:32.310 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2700 IOs, 1050667008 bytes 2023-12-10T19:02:32.450 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2800 IOs, 1079176192 bytes 2023-12-10T19:02:32.645 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2900 IOs, 1150145536 bytes 2023-12-10T19:02:32.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3000 IOs, 1175647232 bytes 2023-12-10T19:02:33.025 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3100 IOs, 1235102720 bytes 2023-12-10T19:02:33.155 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3200 IOs, 1270759424 bytes 2023-12-10T19:02:33.404 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3300 IOs, 1344320512 bytes 2023-12-10T19:02:33.636 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3400 IOs, 1417794560 bytes 2023-12-10T19:02:33.776 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3500 IOs, 1447747584 bytes 2023-12-10T19:02:33.917 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3600 IOs, 1475428352 bytes 2023-12-10T19:02:34.151 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3700 IOs, 1544808448 bytes 2023-12-10T19:02:34.311 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3800 IOs, 1587542016 bytes 2023-12-10T19:02:34.387 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3900 IOs, 1588177920 bytes 2023-12-10T19:02:34.461 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4000 IOs, 1588811776 bytes 2023-12-10T19:02:34.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4100 IOs, 1670995968 bytes 2023-12-10T19:02:34.851 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4200 IOs, 1691231232 bytes 2023-12-10T19:02:34.987 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4300 IOs, 1717378048 bytes 2023-12-10T19:02:35.119 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4400 IOs, 1745687552 bytes 2023-12-10T19:02:35.324 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4500 IOs, 1809675264 bytes 2023-12-10T19:02:35.407 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4600 IOs, 1830822912 bytes 2023-12-10T19:02:35.586 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4700 IOs, 1863885824 bytes 2023-12-10T19:02:35.700 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4800 IOs, 1880471552 bytes 2023-12-10T19:02:35.796 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4900 IOs, 1892427776 bytes 2023-12-10T19:02:35.946 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5000 IOs, 1954705408 bytes 2023-12-10T19:02:36.161 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5100 IOs, 1989449728 bytes 2023-12-10T19:02:36.294 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5200 IOs, 2028389376 bytes 2023-12-10T19:02:36.449 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5300 IOs, 2053266432 bytes 2023-12-10T19:02:36.604 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5400 IOs, 2093255680 bytes 2023-12-10T19:02:36.707 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5500 IOs, 2102432768 bytes 2023-12-10T19:02:36.929 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5600 IOs, 2158787584 bytes 2023-12-10T19:02:37.100 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5700 IOs, 2198941696 bytes 2023-12-10T19:02:37.294 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5800 IOs, 2258545664 bytes 2023-12-10T19:02:37.527 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5900 IOs, 2349399040 bytes 2023-12-10T19:02:37.773 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6000 IOs, 2399347712 bytes 2023-12-10T19:02:37.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6100 IOs, 2465858560 bytes 2023-12-10T19:02:38.194 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6200 IOs, 2528346112 bytes 2023-12-10T19:02:38.404 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6300 IOs, 2597265408 bytes 2023-12-10T19:02:38.652 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6400 IOs, 2687306752 bytes 2023-12-10T19:02:38.893 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6500 IOs, 2771099648 bytes 2023-12-10T19:02:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6600 IOs, 2825729024 bytes 2023-12-10T19:02:39.264 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6700 IOs, 2884728832 bytes 2023-12-10T19:02:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6800 IOs, 2903890944 bytes 2023-12-10T19:02:39.622 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6900 IOs, 2947340288 bytes 2023-12-10T19:02:39.772 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:7000 IOs, 2978250752 bytes 2023-12-10T19:02:39.959 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:7100 IOs, 3026503680 bytes 2023-12-10T19:02:40.120 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:7200 IOs, 3068058624 bytes 2023-12-10T19:02:40.322 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:7300 IOs, 3121165312 bytes 2023-12-10T19:02:40.414 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:7400 IOs, 3121766400 bytes 2023-12-10T19:02:40.520 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:7500 IOs, 3131920384 bytes 2023-12-10T19:02:40.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:7600 IOs, 3174561792 bytes 2023-12-10T19:02:40.728 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.40068405264401874) > weight (0.25) 2023-12-10T19:02:40.729 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.9099634575469913) > weight (0.25) 2023-12-10T19:02:40.729 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.3743205029328759) > weight (0.25) 2023-12-10T19:02:40.729 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.4544934807542932) > weight (0.25) 2023-12-10T19:02:40.729 INFO:tasks.rbd_mirror_thrash:waiting for 66 secs before thrashing 2023-12-10T19:02:40.887 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:7700 IOs, 3224646656 bytes 2023-12-10T19:02:40.989 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=124 2023-12-10T19:02:40.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T19:02:40.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 124 -eq 124 ] 2023-12-10T19:02:40.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:02:40.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_2 2023-12-10T19:02:40.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image_and_enable_mirror cluster2 mirror image_2 snapshot 128M 2023-12-10T19:02:40.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:02:40.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:02:40.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:02:40.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:02:40.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_2 2023-12-10T19:02:40.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:02:40.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:02:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n snapshot ] 2023-12-10T19:02:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:02:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image cluster2 mirror image_2 128M 2023-12-10T19:02:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:02:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:02:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:02:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:02:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_2 2023-12-10T19:02:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:02:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local size=128 2023-12-10T19:02:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n 128M ] 2023-12-10T19:02:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ size=128M 2023-12-10T19:02:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:02:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_2 2023-12-10T19:02:41.038 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ is_pool_mirror_mode_image mirror 2023-12-10T19:02:41.038 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:02:41.038 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ image = image ] 2023-12-10T19:02:41.038 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:02:41.039 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ enable_mirror cluster2 mirror image_2 snapshot 2023-12-10T19:02:41.039 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:02:41.039 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:02:41.039 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_2 2023-12-10T19:02:41.039 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:02:41.039 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 snapshot 2023-12-10T19:02:41.490 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirroring enabled 2023-12-10T19:02:41.500 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2023-12-10T19:02:41.547 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_2': 2023-12-10T19:02:41.547 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:02:41.547 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:02:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 1 2023-12-10T19:02:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 127a5fa662cb 2023-12-10T19:02:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.127a5fa662cb 2023-12-10T19:02:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:02:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering 2023-12-10T19:02:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:02:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:02:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:02:41 2023 2023-12-10T19:02:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:02:41 2023 2023-12-10T19:02:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:02:41 2023 2023-12-10T19:02:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:02:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:02:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: 13cd2888-c5a1-4f79-8bc5-a118e5cace49 2023-12-10T19:02:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: true 2023-12-10T19:02:41.552 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T19:02:41.552 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stress_write_image cluster2 mirror image_2 2023-12-10T19:02:41.552 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:02:41.552 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:02:41.552 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_2 2023-12-10T19:02:41.552 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-10T19:02:41.553 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local duration=9 2023-12-10T19:02:41.554 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T19:02:41.554 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2023-12-10T19:02:41.800 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:100 IOs, 61658112 bytes 2023-12-10T19:02:42.009 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:200 IOs, 128229376 bytes 2023-12-10T19:02:42.109 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:300 IOs, 139139072 bytes 2023-12-10T19:02:42.302 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:400 IOs, 206978048 bytes 2023-12-10T19:02:42.562 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:500 IOs, 283632640 bytes 2023-12-10T19:02:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:600 IOs, 327742464 bytes 2023-12-10T19:02:43.085 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:700 IOs, 445442048 bytes 2023-12-10T19:02:43.295 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:800 IOs, 504312832 bytes 2023-12-10T19:02:43.428 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:900 IOs, 528311296 bytes 2023-12-10T19:02:43.570 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1000 IOs, 556582912 bytes 2023-12-10T19:02:43.726 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1100 IOs, 594875392 bytes 2023-12-10T19:02:43.867 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1200 IOs, 623368192 bytes 2023-12-10T19:02:43.971 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1300 IOs, 638798848 bytes 2023-12-10T19:02:44.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1400 IOs, 675491840 bytes 2023-12-10T19:02:44.278 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1500 IOs, 723266560 bytes 2023-12-10T19:02:44.280 INFO:tasks.ceph.cluster1.osd.2.smithi033.stderr:2023-12-10T19:02:44.276+0000 7fef8ab8e640 -1 reset not still connected to 0x55c59d0c5380 2023-12-10T19:02:44.280 INFO:tasks.ceph.cluster1.osd.2.smithi033.stderr:2023-12-10T19:02:44.276+0000 7fef8ab8e640 -1 reset not still connected to 0x55c59d710c30 2023-12-10T19:02:44.524 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1600 IOs, 807048192 bytes 2023-12-10T19:02:44.627 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1700 IOs, 814705664 bytes 2023-12-10T19:02:44.792 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1800 IOs, 858567680 bytes 2023-12-10T19:02:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1900 IOs, 859203584 bytes 2023-12-10T19:02:45.040 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2000 IOs, 909602816 bytes 2023-12-10T19:02:45.178 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2100 IOs, 947529728 bytes 2023-12-10T19:02:45.429 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2200 IOs, 1016539136 bytes 2023-12-10T19:02:45.641 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2300 IOs, 1063734272 bytes 2023-12-10T19:02:45.776 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2400 IOs, 1088756736 bytes 2023-12-10T19:02:45.894 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2500 IOs, 1107535872 bytes 2023-12-10T19:02:46.045 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2600 IOs, 1147345920 bytes 2023-12-10T19:02:46.181 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2700 IOs, 1170209792 bytes 2023-12-10T19:02:46.291 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2800 IOs, 1186797568 bytes 2023-12-10T19:02:46.376 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2900 IOs, 1187386368 bytes 2023-12-10T19:02:46.563 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3000 IOs, 1257883648 bytes 2023-12-10T19:02:46.759 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3100 IOs, 1325356032 bytes 2023-12-10T19:02:46.921 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3200 IOs, 1347411968 bytes 2023-12-10T19:02:47.025 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3300 IOs, 1351849984 bytes 2023-12-10T19:02:47.192 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3400 IOs, 1400338432 bytes 2023-12-10T19:02:47.354 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3500 IOs, 1439084544 bytes 2023-12-10T19:02:47.503 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3600 IOs, 1474353152 bytes 2023-12-10T19:02:47.646 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3700 IOs, 1494554624 bytes 2023-12-10T19:02:47.746 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3800 IOs, 1504315392 bytes 2023-12-10T19:02:47.907 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3900 IOs, 1547858944 bytes 2023-12-10T19:02:48.033 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4000 IOs, 1570309120 bytes 2023-12-10T19:02:48.189 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4100 IOs, 1612243968 bytes 2023-12-10T19:02:48.353 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4200 IOs, 1663493120 bytes 2023-12-10T19:02:48.597 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4300 IOs, 1731515392 bytes 2023-12-10T19:02:48.710 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4400 IOs, 1741406208 bytes 2023-12-10T19:02:48.872 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4500 IOs, 1779691520 bytes 2023-12-10T19:02:49.039 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4600 IOs, 1820813312 bytes 2023-12-10T19:02:49.191 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4700 IOs, 1856181248 bytes 2023-12-10T19:02:49.382 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4800 IOs, 1914650624 bytes 2023-12-10T19:02:49.652 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4900 IOs, 1994220544 bytes 2023-12-10T19:02:49.836 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5000 IOs, 2046014464 bytes 2023-12-10T19:02:49.976 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5100 IOs, 2093122560 bytes 2023-12-10T19:02:50.210 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5200 IOs, 2135979008 bytes 2023-12-10T19:02:50.302 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5300 IOs, 2136595456 bytes 2023-12-10T19:02:50.486 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5400 IOs, 2182275072 bytes 2023-12-10T19:02:50.556 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=124 2023-12-10T19:02:50.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T19:02:50.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 124 -eq 124 ] 2023-12-10T19:02:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:02:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_3 2023-12-10T19:02:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image_and_enable_mirror cluster2 mirror image_3 snapshot 128M 2023-12-10T19:02:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:02:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:02:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:02:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:02:50.565 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_3 2023-12-10T19:02:50.565 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:02:50.565 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:02:50.565 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n snapshot ] 2023-12-10T19:02:50.565 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:02:50.565 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image cluster2 mirror image_3 128M 2023-12-10T19:02:50.565 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:02:50.565 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:02:50.565 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:02:50.565 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:02:50.565 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_3 2023-12-10T19:02:50.565 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:02:50.566 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local size=128 2023-12-10T19:02:50.566 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n 128M ] 2023-12-10T19:02:50.566 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ size=128M 2023-12-10T19:02:50.566 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:02:50.566 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_3 2023-12-10T19:02:50.608 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ is_pool_mirror_mode_image mirror 2023-12-10T19:02:50.608 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:02:50.608 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ image = image ] 2023-12-10T19:02:50.608 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:02:50.608 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ enable_mirror cluster2 mirror image_3 snapshot 2023-12-10T19:02:50.608 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:02:50.608 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:02:50.608 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_3 2023-12-10T19:02:50.608 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:02:50.608 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 snapshot 2023-12-10T19:02:51.419 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirroring enabled 2023-12-10T19:02:51.430 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2023-12-10T19:02:51.476 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_3': 2023-12-10T19:02:51.476 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:02:51.476 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:02:51.476 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 1 2023-12-10T19:02:51.476 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 1286cb96b8a1 2023-12-10T19:02:51.476 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.1286cb96b8a1 2023-12-10T19:02:51.476 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:02:51.476 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering 2023-12-10T19:02:51.476 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:02:51.476 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:02:51.477 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:02:50 2023 2023-12-10T19:02:51.477 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:02:50 2023 2023-12-10T19:02:51.477 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:02:50 2023 2023-12-10T19:02:51.477 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:02:51.477 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:02:51.477 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: 3140c852-fad6-4a00-a40c-46869faa3ea1 2023-12-10T19:02:51.477 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: true 2023-12-10T19:02:51.480 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T19:02:51.480 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stress_write_image cluster2 mirror image_3 2023-12-10T19:02:51.480 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:02:51.480 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:02:51.480 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_3 2023-12-10T19:02:51.481 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-10T19:02:51.482 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local duration=12 2023-12-10T19:02:51.482 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T19:02:51.482 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2023-12-10T19:02:51.836 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:100 IOs, 99184640 bytes 2023-12-10T19:02:51.918 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:200 IOs, 99763200 bytes 2023-12-10T19:02:52.031 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:300 IOs, 123541504 bytes 2023-12-10T19:02:52.152 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:400 IOs, 138035200 bytes 2023-12-10T19:02:52.310 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:500 IOs, 175292416 bytes 2023-12-10T19:02:52.438 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:600 IOs, 204627968 bytes 2023-12-10T19:02:52.701 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:700 IOs, 291459072 bytes 2023-12-10T19:02:52.884 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:800 IOs, 351600640 bytes 2023-12-10T19:02:53.088 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:900 IOs, 415212544 bytes 2023-12-10T19:02:53.206 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1000 IOs, 426869760 bytes 2023-12-10T19:02:53.296 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1100 IOs, 432612352 bytes 2023-12-10T19:02:53.452 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1200 IOs, 469771264 bytes 2023-12-10T19:02:53.636 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1300 IOs, 522695680 bytes 2023-12-10T19:02:53.778 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1400 IOs, 550241280 bytes 2023-12-10T19:02:53.914 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1500 IOs, 580501504 bytes 2023-12-10T19:02:54.125 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1600 IOs, 646048768 bytes 2023-12-10T19:02:54.282 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1700 IOs, 684241920 bytes 2023-12-10T19:02:54.455 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1800 IOs, 737789952 bytes 2023-12-10T19:02:54.719 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1900 IOs, 823789568 bytes 2023-12-10T19:02:54.877 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2000 IOs, 859455488 bytes 2023-12-10T19:02:55.042 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2100 IOs, 902838272 bytes 2023-12-10T19:02:55.172 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2200 IOs, 929150976 bytes 2023-12-10T19:02:55.360 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2300 IOs, 982447104 bytes 2023-12-10T19:02:55.435 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2400 IOs, 983068672 bytes 2023-12-10T19:02:55.543 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2500 IOs, 992031744 bytes 2023-12-10T19:02:55.744 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2600 IOs, 1049757696 bytes 2023-12-10T19:02:55.894 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2700 IOs, 1076533248 bytes 2023-12-10T19:02:56.089 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2800 IOs, 1128977408 bytes 2023-12-10T19:02:56.245 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2900 IOs, 1164503040 bytes 2023-12-10T19:02:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3000 IOs, 1244125184 bytes 2023-12-10T19:02:56.648 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3100 IOs, 1291298816 bytes 2023-12-10T19:02:56.849 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3200 IOs, 1333639168 bytes 2023-12-10T19:02:57.040 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3300 IOs, 1386076160 bytes 2023-12-10T19:02:57.221 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3400 IOs, 1440868352 bytes 2023-12-10T19:02:57.361 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3500 IOs, 1471047680 bytes 2023-12-10T19:02:57.444 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3600 IOs, 1477341184 bytes 2023-12-10T19:02:57.600 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3700 IOs, 1538224128 bytes 2023-12-10T19:02:57.810 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3800 IOs, 1573976064 bytes 2023-12-10T19:02:58.063 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3900 IOs, 1650236416 bytes 2023-12-10T19:02:58.292 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4000 IOs, 1713019904 bytes 2023-12-10T19:02:58.446 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4100 IOs, 1747040256 bytes 2023-12-10T19:02:58.525 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4200 IOs, 1747652608 bytes 2023-12-10T19:02:58.685 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4300 IOs, 1788408832 bytes 2023-12-10T19:02:58.854 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4400 IOs, 1821872128 bytes 2023-12-10T19:02:58.961 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4500 IOs, 1837301760 bytes 2023-12-10T19:02:59.108 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4600 IOs, 1874567168 bytes 2023-12-10T19:02:59.422 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4700 IOs, 1977741312 bytes 2023-12-10T19:02:59.680 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4800 IOs, 2063875072 bytes 2023-12-10T19:02:59.886 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4900 IOs, 2128323584 bytes 2023-12-10T19:03:00.010 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5000 IOs, 2144466944 bytes 2023-12-10T19:03:00.285 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5100 IOs, 2238373888 bytes 2023-12-10T19:03:00.369 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5200 IOs, 2239376384 bytes 2023-12-10T19:03:00.476 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5300 IOs, 2253773824 bytes 2023-12-10T19:03:00.745 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5400 IOs, 2345068544 bytes 2023-12-10T19:03:00.865 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5500 IOs, 2360518656 bytes 2023-12-10T19:03:00.940 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5600 IOs, 2361142272 bytes 2023-12-10T19:03:01.143 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5700 IOs, 2412111872 bytes 2023-12-10T19:03:01.396 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5800 IOs, 2515652608 bytes 2023-12-10T19:03:01.589 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5900 IOs, 2553116672 bytes 2023-12-10T19:03:01.809 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6000 IOs, 2623811584 bytes 2023-12-10T19:03:01.925 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6100 IOs, 2639978496 bytes 2023-12-10T19:03:02.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6200 IOs, 2697036800 bytes 2023-12-10T19:03:02.245 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6300 IOs, 2731671552 bytes 2023-12-10T19:03:02.388 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6400 IOs, 2751919104 bytes 2023-12-10T19:03:02.527 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6500 IOs, 2787766272 bytes 2023-12-10T19:03:02.664 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6600 IOs, 2812661760 bytes 2023-12-10T19:03:02.860 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6700 IOs, 2861806592 bytes 2023-12-10T19:03:02.987 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6800 IOs, 2886556672 bytes 2023-12-10T19:03:03.190 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6900 IOs, 2947332096 bytes 2023-12-10T19:03:03.282 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:7000 IOs, 2955681792 bytes 2023-12-10T19:03:03.486 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=124 2023-12-10T19:03:03.486 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T19:03:03.486 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 124 -eq 124 ] 2023-12-10T19:03:03.486 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:03:03.486 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_4 2023-12-10T19:03:03.486 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image_and_enable_mirror cluster2 mirror image_4 snapshot 128M 2023-12-10T19:03:03.486 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:03:03.486 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:03:03.486 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:03:03.487 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:03:03.487 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_4 2023-12-10T19:03:03.487 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:03:03.487 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:03:03.487 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n snapshot ] 2023-12-10T19:03:03.487 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:03:03.487 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image cluster2 mirror image_4 128M 2023-12-10T19:03:03.487 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:03:03.487 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:03:03.487 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:03:03.487 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:03:03.487 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_4 2023-12-10T19:03:03.487 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:03:03.487 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local size=128 2023-12-10T19:03:03.487 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n 128M ] 2023-12-10T19:03:03.488 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ size=128M 2023-12-10T19:03:03.488 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:03:03.488 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_4 2023-12-10T19:03:03.534 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ is_pool_mirror_mode_image mirror 2023-12-10T19:03:03.535 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:03:03.535 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ image = image ] 2023-12-10T19:03:03.535 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:03:03.535 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ enable_mirror cluster2 mirror image_4 snapshot 2023-12-10T19:03:03.535 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:03:03.535 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:03:03.535 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_4 2023-12-10T19:03:03.535 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:03:03.535 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 snapshot 2023-12-10T19:03:04.453 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirroring enabled 2023-12-10T19:03:04.462 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2023-12-10T19:03:04.510 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_4': 2023-12-10T19:03:04.510 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:03:04.510 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:03:04.510 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 1 2023-12-10T19:03:04.510 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 129287060c48 2023-12-10T19:03:04.510 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.129287060c48 2023-12-10T19:03:04.510 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:03:04.510 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering 2023-12-10T19:03:04.510 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:03:04.510 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:03:04.511 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:03:03 2023 2023-12-10T19:03:04.511 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:03:03 2023 2023-12-10T19:03:04.511 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:03:03 2023 2023-12-10T19:03:04.511 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:03:04.511 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:03:04.511 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: 2e46723e-74b1-4e29-b6e7-8617064dd725 2023-12-10T19:03:04.511 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: true 2023-12-10T19:03:04.513 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T19:03:04.513 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stress_write_image cluster2 mirror image_4 2023-12-10T19:03:04.513 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:03:04.513 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:03:04.513 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_4 2023-12-10T19:03:04.513 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-10T19:03:04.515 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local duration=14 2023-12-10T19:03:04.515 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T19:03:04.515 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2023-12-10T19:03:04.839 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:100 IOs, 87290880 bytes 2023-12-10T19:03:05.023 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:200 IOs, 142776320 bytes 2023-12-10T19:03:05.226 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:300 IOs, 177488896 bytes 2023-12-10T19:03:05.440 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:400 IOs, 241253376 bytes 2023-12-10T19:03:05.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:500 IOs, 319082496 bytes 2023-12-10T19:03:05.921 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:600 IOs, 397631488 bytes 2023-12-10T19:03:06.066 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:700 IOs, 428141568 bytes 2023-12-10T19:03:06.208 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:800 IOs, 454558720 bytes 2023-12-10T19:03:06.391 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:900 IOs, 516545536 bytes 2023-12-10T19:03:06.559 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1000 IOs, 534110208 bytes 2023-12-10T19:03:06.647 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1100 IOs, 538665984 bytes 2023-12-10T19:03:06.797 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1200 IOs, 569381888 bytes 2023-12-10T19:03:06.926 INFO:tasks.ceph.cluster2.osd.2.smithi076.stderr:2023-12-10T19:03:06.927+0000 7f7f6add3640 -1 reset not still connected to 0x56430b819d40 2023-12-10T19:03:06.928 INFO:tasks.ceph.cluster2.osd.2.smithi076.stderr:2023-12-10T19:03:06.931+0000 7f7f6add3640 -1 reset not still connected to 0x56430fcbf6c0 2023-12-10T19:03:06.928 INFO:tasks.ceph.cluster2.osd.2.smithi076.stderr:2023-12-10T19:03:06.931+0000 7f7f6add3640 -1 reset not still connected to 0x564313584820 2023-12-10T19:03:06.928 INFO:tasks.ceph.cluster2.osd.2.smithi076.stderr:2023-12-10T19:03:06.931+0000 7f7f6add3640 -1 reset not still connected to 0x5643136d5110 2023-12-10T19:03:06.928 INFO:tasks.ceph.cluster2.osd.2.smithi076.stderr:2023-12-10T19:03:06.931+0000 7f7f6add3640 -1 reset not still connected to 0x56431704cb60 2023-12-10T19:03:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1300 IOs, 633864192 bytes 2023-12-10T19:03:07.203 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1400 IOs, 682440704 bytes 2023-12-10T19:03:07.387 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1500 IOs, 731239424 bytes 2023-12-10T19:03:07.668 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1600 IOs, 832905216 bytes 2023-12-10T19:03:07.774 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1700 IOs, 833528832 bytes 2023-12-10T19:03:07.902 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1800 IOs, 853436416 bytes 2023-12-10T19:03:08.050 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1900 IOs, 889133056 bytes 2023-12-10T19:03:08.200 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2000 IOs, 913303552 bytes 2023-12-10T19:03:08.418 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2100 IOs, 979553280 bytes 2023-12-10T19:03:08.548 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2200 IOs, 1001646080 bytes 2023-12-10T19:03:08.746 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2300 IOs, 1067324416 bytes 2023-12-10T19:03:08.948 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2400 IOs, 1115937792 bytes 2023-12-10T19:03:09.101 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2500 IOs, 1154313216 bytes 2023-12-10T19:03:09.215 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2600 IOs, 1159025664 bytes 2023-12-10T19:03:09.395 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2700 IOs, 1209439232 bytes 2023-12-10T19:03:09.568 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2800 IOs, 1256738816 bytes 2023-12-10T19:03:09.746 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2900 IOs, 1308368896 bytes 2023-12-10T19:03:09.885 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3000 IOs, 1328896000 bytes 2023-12-10T19:03:10.005 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3100 IOs, 1354860544 bytes 2023-12-10T19:03:10.128 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3200 IOs, 1368084480 bytes 2023-12-10T19:03:10.403 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3300 IOs, 1452119040 bytes 2023-12-10T19:03:10.548 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3400 IOs, 1481507840 bytes 2023-12-10T19:03:10.705 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3500 IOs, 1511449600 bytes 2023-12-10T19:03:10.885 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3600 IOs, 1558223872 bytes 2023-12-10T19:03:11.089 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3700 IOs, 1637361664 bytes 2023-12-10T19:03:11.195 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3800 IOs, 1638977536 bytes 2023-12-10T19:03:11.362 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3900 IOs, 1676222464 bytes 2023-12-10T19:03:11.561 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4000 IOs, 1734651904 bytes 2023-12-10T19:03:11.691 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4100 IOs, 1754384384 bytes 2023-12-10T19:03:11.858 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4200 IOs, 1804098560 bytes 2023-12-10T19:03:11.974 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4300 IOs, 1820166144 bytes 2023-12-10T19:03:12.070 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4400 IOs, 1826227200 bytes 2023-12-10T19:03:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4500 IOs, 1876255744 bytes 2023-12-10T19:03:12.626 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4600 IOs, 1994017792 bytes 2023-12-10T19:03:12.844 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4700 IOs, 2050649088 bytes 2023-12-10T19:03:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4800 IOs, 2113987584 bytes 2023-12-10T19:03:13.223 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4900 IOs, 2136471552 bytes 2023-12-10T19:03:13.380 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5000 IOs, 2164471808 bytes 2023-12-10T19:03:13.595 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5100 IOs, 2223408128 bytes 2023-12-10T19:03:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5200 IOs, 2255582208 bytes 2023-12-10T19:03:13.925 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5300 IOs, 2298948608 bytes 2023-12-10T19:03:14.016 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5400 IOs, 2299572224 bytes 2023-12-10T19:03:14.288 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5500 IOs, 2386265088 bytes 2023-12-10T19:03:14.491 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5600 IOs, 2445360128 bytes 2023-12-10T19:03:14.666 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5700 IOs, 2485002240 bytes 2023-12-10T19:03:14.834 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5800 IOs, 2531515392 bytes 2023-12-10T19:03:15.012 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5900 IOs, 2572905472 bytes 2023-12-10T19:03:15.220 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6000 IOs, 2629751808 bytes 2023-12-10T19:03:15.511 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6100 IOs, 2727798784 bytes 2023-12-10T19:03:15.696 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6200 IOs, 2771857408 bytes 2023-12-10T19:03:15.885 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6300 IOs, 2819307520 bytes 2023-12-10T19:03:16.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6400 IOs, 2899360768 bytes 2023-12-10T19:03:16.277 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6500 IOs, 2922328064 bytes 2023-12-10T19:03:16.377 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6600 IOs, 2922933248 bytes 2023-12-10T19:03:16.510 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6700 IOs, 2947954688 bytes 2023-12-10T19:03:16.750 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6800 IOs, 3012445184 bytes 2023-12-10T19:03:16.937 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6900 IOs, 3061213184 bytes 2023-12-10T19:03:17.108 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:7000 IOs, 3098108928 bytes 2023-12-10T19:03:17.255 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:7100 IOs, 3128221696 bytes 2023-12-10T19:03:17.388 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:7200 IOs, 3163044864 bytes 2023-12-10T19:03:17.514 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:7300 IOs, 3171487744 bytes 2023-12-10T19:03:17.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:7400 IOs, 3239963648 bytes 2023-12-10T19:03:17.856 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:7500 IOs, 3248098304 bytes 2023-12-10T19:03:18.057 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:7600 IOs, 3302449152 bytes 2023-12-10T19:03:18.170 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:7700 IOs, 3316173824 bytes 2023-12-10T19:03:18.371 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:7800 IOs, 3375873024 bytes 2023-12-10T19:03:18.518 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=124 2023-12-10T19:03:18.519 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T19:03:18.519 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 124 -eq 124 ] 2023-12-10T19:03:18.519 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:03:18.519 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_5 2023-12-10T19:03:18.519 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image_and_enable_mirror cluster2 mirror image_5 snapshot 128M 2023-12-10T19:03:18.519 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:03:18.519 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:03:18.519 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:03:18.519 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:03:18.519 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_5 2023-12-10T19:03:18.519 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:03:18.519 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:03:18.520 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n snapshot ] 2023-12-10T19:03:18.520 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:03:18.520 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image cluster2 mirror image_5 128M 2023-12-10T19:03:18.520 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:03:18.520 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:03:18.520 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:03:18.520 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:03:18.520 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_5 2023-12-10T19:03:18.520 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:03:18.520 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local size=128 2023-12-10T19:03:18.520 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n 128M ] 2023-12-10T19:03:18.520 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ size=128M 2023-12-10T19:03:18.520 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:03:18.521 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_5 2023-12-10T19:03:18.567 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ is_pool_mirror_mode_image mirror 2023-12-10T19:03:18.567 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:03:18.567 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ image = image ] 2023-12-10T19:03:18.568 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:03:18.568 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ enable_mirror cluster2 mirror image_5 snapshot 2023-12-10T19:03:18.568 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:03:18.568 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:03:18.568 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_5 2023-12-10T19:03:18.568 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:03:18.568 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 snapshot 2023-12-10T19:03:19.419 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirroring enabled 2023-12-10T19:03:19.430 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2023-12-10T19:03:19.474 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_5': 2023-12-10T19:03:19.474 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:03:19.474 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:03:19.474 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 1 2023-12-10T19:03:19.474 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 129ef8d42b97 2023-12-10T19:03:19.474 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.129ef8d42b97 2023-12-10T19:03:19.474 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:03:19.474 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering 2023-12-10T19:03:19.474 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:03:19.474 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:03:19.475 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:03:18 2023 2023-12-10T19:03:19.475 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:03:18 2023 2023-12-10T19:03:19.475 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:03:18 2023 2023-12-10T19:03:19.475 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:03:19.475 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:03:19.475 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: c911f848-6f73-4853-a1d8-243e89af66d6 2023-12-10T19:03:19.475 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: true 2023-12-10T19:03:19.478 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T19:03:19.478 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stress_write_image cluster2 mirror image_5 2023-12-10T19:03:19.478 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:03:19.478 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:03:19.478 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_5 2023-12-10T19:03:19.478 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-10T19:03:19.479 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local duration=11 2023-12-10T19:03:19.480 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T19:03:19.480 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2023-12-10T19:03:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:100 IOs, 93199360 bytes 2023-12-10T19:03:19.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:200 IOs, 136880128 bytes 2023-12-10T19:03:20.159 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:300 IOs, 205694976 bytes 2023-12-10T19:03:20.364 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:400 IOs, 261128192 bytes 2023-12-10T19:03:20.559 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:500 IOs, 311981056 bytes 2023-12-10T19:03:20.684 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:600 IOs, 340379648 bytes 2023-12-10T19:03:20.917 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:700 IOs, 396383232 bytes 2023-12-10T19:03:21.087 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:800 IOs, 436429824 bytes 2023-12-10T19:03:21.261 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:900 IOs, 470960128 bytes 2023-12-10T19:03:21.351 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1000 IOs, 473565184 bytes 2023-12-10T19:03:21.455 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1100 IOs, 474191872 bytes 2023-12-10T19:03:21.602 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1200 IOs, 500311040 bytes 2023-12-10T19:03:21.723 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1300 IOs, 513277952 bytes 2023-12-10T19:03:21.927 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1400 IOs, 562779136 bytes 2023-12-10T19:03:22.019 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1500 IOs, 563378176 bytes 2023-12-10T19:03:22.145 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1600 IOs, 580639744 bytes 2023-12-10T19:03:22.307 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1700 IOs, 615888896 bytes 2023-12-10T19:03:22.485 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1800 IOs, 659508224 bytes 2023-12-10T19:03:22.637 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1900 IOs, 687355904 bytes 2023-12-10T19:03:22.775 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2000 IOs, 706262016 bytes 2023-12-10T19:03:22.875 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2100 IOs, 725132288 bytes 2023-12-10T19:03:23.221 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2200 IOs, 832306176 bytes 2023-12-10T19:03:23.357 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2300 IOs, 846969856 bytes 2023-12-10T19:03:23.537 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2400 IOs, 882476032 bytes 2023-12-10T19:03:23.681 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2500 IOs, 923744256 bytes 2023-12-10T19:03:23.875 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2600 IOs, 954151936 bytes 2023-12-10T19:03:24.105 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2700 IOs, 1023908864 bytes 2023-12-10T19:03:24.284 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2800 IOs, 1049593856 bytes 2023-12-10T19:03:24.389 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2900 IOs, 1050194944 bytes 2023-12-10T19:03:24.502 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3000 IOs, 1057195008 bytes 2023-12-10T19:03:24.649 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3100 IOs, 1084439552 bytes 2023-12-10T19:03:24.882 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3200 IOs, 1138724864 bytes 2023-12-10T19:03:25.193 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3300 IOs, 1231371264 bytes 2023-12-10T19:03:25.374 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3400 IOs, 1265566720 bytes 2023-12-10T19:03:25.555 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3500 IOs, 1304584192 bytes 2023-12-10T19:03:25.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3600 IOs, 1323573248 bytes 2023-12-10T19:03:25.876 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3700 IOs, 1364415488 bytes 2023-12-10T19:03:26.034 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3800 IOs, 1397143552 bytes 2023-12-10T19:03:26.224 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3900 IOs, 1447879680 bytes 2023-12-10T19:03:26.420 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4000 IOs, 1493744640 bytes 2023-12-10T19:03:26.526 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4100 IOs, 1504079872 bytes 2023-12-10T19:03:26.735 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4200 IOs, 1562253312 bytes 2023-12-10T19:03:26.948 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4300 IOs, 1623413760 bytes 2023-12-10T19:03:27.100 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4400 IOs, 1648600064 bytes 2023-12-10T19:03:27.301 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4500 IOs, 1711541248 bytes 2023-12-10T19:03:27.489 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4600 IOs, 1758771200 bytes 2023-12-10T19:03:27.588 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4700 IOs, 1759404032 bytes 2023-12-10T19:03:27.746 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4800 IOs, 1790091264 bytes 2023-12-10T19:03:27.931 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4900 IOs, 1838820352 bytes 2023-12-10T19:03:28.121 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5000 IOs, 1895361536 bytes 2023-12-10T19:03:28.220 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5100 IOs, 1903155200 bytes 2023-12-10T19:03:28.327 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5200 IOs, 1911263232 bytes 2023-12-10T19:03:28.467 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5300 IOs, 1939724288 bytes 2023-12-10T19:03:28.696 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5400 IOs, 2011420672 bytes 2023-12-10T19:03:28.888 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5500 IOs, 2053838848 bytes 2023-12-10T19:03:29.085 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5600 IOs, 2114223104 bytes 2023-12-10T19:03:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5700 IOs, 2118063104 bytes 2023-12-10T19:03:29.389 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5800 IOs, 2150262784 bytes 2023-12-10T19:03:29.627 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5900 IOs, 2220396544 bytes 2023-12-10T19:03:29.782 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6000 IOs, 2252056576 bytes 2023-12-10T19:03:29.874 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6100 IOs, 2252666880 bytes 2023-12-10T19:03:30.073 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6200 IOs, 2316040192 bytes 2023-12-10T19:03:30.281 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6300 IOs, 2363713536 bytes 2023-12-10T19:03:30.415 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6400 IOs, 2381380608 bytes 2023-12-10T19:03:30.484 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=124 2023-12-10T19:03:30.484 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T19:03:30.484 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 124 -eq 124 ] 2023-12-10T19:03:30.484 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:03:30.484 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_6 2023-12-10T19:03:30.484 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image_and_enable_mirror cluster2 mirror image_6 snapshot 128M 2023-12-10T19:03:30.484 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:03:30.484 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:03:30.485 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:03:30.485 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:03:30.485 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_6 2023-12-10T19:03:30.485 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:03:30.485 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:03:30.485 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n snapshot ] 2023-12-10T19:03:30.485 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:03:30.485 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image cluster2 mirror image_6 128M 2023-12-10T19:03:30.485 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:03:30.485 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:03:30.485 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:03:30.485 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:03:30.485 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_6 2023-12-10T19:03:30.485 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:03:30.485 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local size=128 2023-12-10T19:03:30.485 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n 128M ] 2023-12-10T19:03:30.486 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ size=128M 2023-12-10T19:03:30.486 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:03:30.486 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_6 2023-12-10T19:03:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ is_pool_mirror_mode_image mirror 2023-12-10T19:03:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:03:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ image = image ] 2023-12-10T19:03:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:03:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ enable_mirror cluster2 mirror image_6 snapshot 2023-12-10T19:03:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:03:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:03:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_6 2023-12-10T19:03:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:03:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 snapshot 2023-12-10T19:03:31.432 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirroring enabled 2023-12-10T19:03:31.444 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2023-12-10T19:03:31.490 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_6': 2023-12-10T19:03:31.490 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:03:31.491 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:03:31.491 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 1 2023-12-10T19:03:31.491 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 12aab67b48dd 2023-12-10T19:03:31.491 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.12aab67b48dd 2023-12-10T19:03:31.491 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:03:31.491 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering 2023-12-10T19:03:31.491 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:03:31.491 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:03:31.491 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:03:30 2023 2023-12-10T19:03:31.491 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:03:30 2023 2023-12-10T19:03:31.491 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:03:30 2023 2023-12-10T19:03:31.491 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:03:31.491 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:03:31.491 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: c1f7c734-b9ec-4996-95de-10380a5d720f 2023-12-10T19:03:31.491 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: true 2023-12-10T19:03:31.495 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T19:03:31.495 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stress_write_image cluster2 mirror image_6 2023-12-10T19:03:31.495 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:03:31.495 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:03:31.495 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_6 2023-12-10T19:03:31.495 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-10T19:03:31.496 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local duration=14 2023-12-10T19:03:31.496 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T19:03:31.497 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2023-12-10T19:03:31.781 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:100 IOs, 81786880 bytes 2023-12-10T19:03:31.964 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:200 IOs, 127753216 bytes 2023-12-10T19:03:32.192 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:300 IOs, 188377088 bytes 2023-12-10T19:03:32.321 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:400 IOs, 208057344 bytes 2023-12-10T19:03:32.525 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:500 IOs, 260406272 bytes 2023-12-10T19:03:32.639 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:600 IOs, 267176960 bytes 2023-12-10T19:03:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:700 IOs, 294189056 bytes 2023-12-10T19:03:32.963 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:800 IOs, 335879168 bytes 2023-12-10T19:03:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:900 IOs, 394285056 bytes 2023-12-10T19:03:33.285 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1000 IOs, 411265024 bytes 2023-12-10T19:03:33.493 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1100 IOs, 475036672 bytes 2023-12-10T19:03:33.627 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1200 IOs, 486837248 bytes 2023-12-10T19:03:33.734 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1300 IOs, 494436352 bytes 2023-12-10T19:03:33.914 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1400 IOs, 548566016 bytes 2023-12-10T19:03:34.235 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1500 IOs, 626418688 bytes 2023-12-10T19:03:34.423 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1600 IOs, 675593216 bytes 2023-12-10T19:03:34.529 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1700 IOs, 679635968 bytes 2023-12-10T19:03:34.669 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1800 IOs, 703465472 bytes 2023-12-10T19:03:34.880 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1900 IOs, 767275008 bytes 2023-12-10T19:03:35.028 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2000 IOs, 796030976 bytes 2023-12-10T19:03:35.136 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2100 IOs, 807881728 bytes 2023-12-10T19:03:35.364 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2200 IOs, 880729088 bytes 2023-12-10T19:03:35.507 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2300 IOs, 912224256 bytes 2023-12-10T19:03:35.726 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2400 IOs, 983187456 bytes 2023-12-10T19:03:35.915 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2500 IOs, 1026954240 bytes 2023-12-10T19:03:36.067 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2600 IOs, 1091173376 bytes 2023-12-10T19:03:36.238 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2700 IOs, 1094682624 bytes 2023-12-10T19:03:36.332 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2800 IOs, 1097401344 bytes 2023-12-10T19:03:36.539 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2900 IOs, 1156201472 bytes 2023-12-10T19:03:36.689 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3000 IOs, 1196846080 bytes 2023-12-10T19:03:36.814 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3100 IOs, 1204130816 bytes 2023-12-10T19:03:36.968 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3200 IOs, 1222802432 bytes 2023-12-10T19:03:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3300 IOs, 1258667008 bytes 2023-12-10T19:03:37.492 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3400 IOs, 1293983744 bytes 2023-12-10T19:03:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3500 IOs, 1327814656 bytes 2023-12-10T19:03:37.841 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3600 IOs, 1365319680 bytes 2023-12-10T19:03:38.005 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3700 IOs, 1393644544 bytes 2023-12-10T19:03:38.258 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3800 IOs, 1473327104 bytes 2023-12-10T19:03:38.364 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3900 IOs, 1481023488 bytes 2023-12-10T19:03:38.581 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4000 IOs, 1535723520 bytes 2023-12-10T19:03:38.730 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.27750153990562) > weight (0.25) 2023-12-10T19:03:38.730 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.9552495524631899) > weight (0.25) 2023-12-10T19:03:38.730 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.563677908536872) > weight (0.25) 2023-12-10T19:03:38.730 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.923964596629675) > weight (0.25) 2023-12-10T19:03:38.730 INFO:tasks.rbd_mirror_thrash:waiting for 90 secs before thrashing 2023-12-10T19:03:38.752 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4100 IOs, 1574445056 bytes 2023-12-10T19:03:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4200 IOs, 1618808832 bytes 2023-12-10T19:03:39.110 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4300 IOs, 1656702976 bytes 2023-12-10T19:03:39.284 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4400 IOs, 1691052032 bytes 2023-12-10T19:03:39.484 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4500 IOs, 1736755200 bytes 2023-12-10T19:03:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4600 IOs, 1806755840 bytes 2023-12-10T19:03:39.865 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4700 IOs, 1830630400 bytes 2023-12-10T19:03:40.122 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4800 IOs, 1910223872 bytes 2023-12-10T19:03:40.229 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4900 IOs, 1916279808 bytes 2023-12-10T19:03:40.338 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5000 IOs, 1922612224 bytes 2023-12-10T19:03:40.455 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5100 IOs, 1928536064 bytes 2023-12-10T19:03:40.641 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5200 IOs, 1961527296 bytes 2023-12-10T19:03:40.834 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5300 IOs, 2007559168 bytes 2023-12-10T19:03:40.987 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5400 IOs, 2045265920 bytes 2023-12-10T19:03:41.416 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5500 IOs, 2201115648 bytes 2023-12-10T19:03:41.632 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5600 IOs, 2264216576 bytes 2023-12-10T19:03:41.801 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5700 IOs, 2302895104 bytes 2023-12-10T19:03:42.052 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5800 IOs, 2375896064 bytes 2023-12-10T19:03:42.155 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5900 IOs, 2376500224 bytes 2023-12-10T19:03:42.366 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6000 IOs, 2434476032 bytes 2023-12-10T19:03:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6100 IOs, 2491212800 bytes 2023-12-10T19:03:42.754 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6200 IOs, 2535089152 bytes 2023-12-10T19:03:42.988 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6300 IOs, 2602620928 bytes 2023-12-10T19:03:43.114 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6400 IOs, 2616064000 bytes 2023-12-10T19:03:43.344 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6500 IOs, 2696689664 bytes 2023-12-10T19:03:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6600 IOs, 2730974208 bytes 2023-12-10T19:03:43.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6700 IOs, 2802188288 bytes 2023-12-10T19:03:43.844 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6800 IOs, 2802806784 bytes 2023-12-10T19:03:43.965 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6900 IOs, 2823192576 bytes 2023-12-10T19:03:44.099 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:7000 IOs, 2857077760 bytes 2023-12-10T19:03:44.268 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:7100 IOs, 2882309120 bytes 2023-12-10T19:03:44.425 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:7200 IOs, 2920370176 bytes 2023-12-10T19:03:44.619 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:7300 IOs, 2969785344 bytes 2023-12-10T19:03:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:7400 IOs, 3083198464 bytes 2023-12-10T19:03:45.060 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:7500 IOs, 3099549696 bytes 2023-12-10T19:03:45.340 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:7600 IOs, 3179641856 bytes 2023-12-10T19:03:45.501 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=124 2023-12-10T19:03:45.501 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T19:03:45.501 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 124 -eq 124 ] 2023-12-10T19:03:45.501 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:03:45.501 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_7 2023-12-10T19:03:45.501 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image_and_enable_mirror cluster2 mirror image_7 snapshot 128M 2023-12-10T19:03:45.501 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:03:45.501 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:03:45.501 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:03:45.501 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:03:45.502 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_7 2023-12-10T19:03:45.502 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:03:45.502 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:03:45.502 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n snapshot ] 2023-12-10T19:03:45.502 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:03:45.502 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image cluster2 mirror image_7 128M 2023-12-10T19:03:45.502 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:03:45.502 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:03:45.502 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:03:45.502 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:03:45.502 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_7 2023-12-10T19:03:45.502 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:03:45.502 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local size=128 2023-12-10T19:03:45.502 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n 128M ] 2023-12-10T19:03:45.502 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ size=128M 2023-12-10T19:03:45.503 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:03:45.503 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_7 2023-12-10T19:03:45.549 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ is_pool_mirror_mode_image mirror 2023-12-10T19:03:45.549 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:03:45.549 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ image = image ] 2023-12-10T19:03:45.549 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:03:45.549 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ enable_mirror cluster2 mirror image_7 snapshot 2023-12-10T19:03:45.550 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:03:45.550 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:03:45.550 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_7 2023-12-10T19:03:45.550 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:03:45.550 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 snapshot 2023-12-10T19:03:45.908 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirroring enabled 2023-12-10T19:03:45.920 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2023-12-10T19:03:45.966 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_7': 2023-12-10T19:03:45.966 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:03:45.966 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:03:45.966 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 1 2023-12-10T19:03:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 12b6bd0dab6 2023-12-10T19:03:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.12b6bd0dab6 2023-12-10T19:03:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:03:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering 2023-12-10T19:03:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:03:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:03:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:03:45 2023 2023-12-10T19:03:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:03:45 2023 2023-12-10T19:03:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:03:45 2023 2023-12-10T19:03:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:03:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:03:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: f8539d7e-11cd-4f60-9d51-e1da51589906 2023-12-10T19:03:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: true 2023-12-10T19:03:45.971 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T19:03:45.971 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stress_write_image cluster2 mirror image_7 2023-12-10T19:03:45.971 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:03:45.971 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:03:45.971 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_7 2023-12-10T19:03:45.971 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-10T19:03:45.972 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local duration=12 2023-12-10T19:03:45.972 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T19:03:45.972 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2023-12-10T19:03:46.208 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:100 IOs, 42576896 bytes 2023-12-10T19:03:46.386 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:200 IOs, 87547904 bytes 2023-12-10T19:03:46.566 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:300 IOs, 141853696 bytes 2023-12-10T19:03:46.729 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.3993362505523398) > weight (0.25) 2023-12-10T19:03:46.730 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.8386555638263705) > weight (0.25) 2023-12-10T19:03:46.730 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.6595071149312878) > weight (0.25) 2023-12-10T19:03:46.730 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.47317237518417854) > weight (0.25) 2023-12-10T19:03:46.730 INFO:tasks.rbd_mirror_thrash:waiting for 74 secs before thrashing 2023-12-10T19:03:46.793 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:400 IOs, 197515264 bytes 2023-12-10T19:03:46.928 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:500 IOs, 217440256 bytes 2023-12-10T19:03:47.038 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:600 IOs, 231357440 bytes 2023-12-10T19:03:47.160 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:700 IOs, 231980032 bytes 2023-12-10T19:03:47.313 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:800 IOs, 265400320 bytes 2023-12-10T19:03:47.571 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:900 IOs, 346608640 bytes 2023-12-10T19:03:47.682 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1000 IOs, 353155072 bytes 2023-12-10T19:03:47.863 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1100 IOs, 398209024 bytes 2023-12-10T19:03:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1200 IOs, 497089536 bytes 2023-12-10T19:03:48.314 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1300 IOs, 537298944 bytes 2023-12-10T19:03:48.451 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1400 IOs, 562344960 bytes 2023-12-10T19:03:48.659 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1500 IOs, 618510336 bytes 2023-12-10T19:03:48.793 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1600 IOs, 636438528 bytes 2023-12-10T19:03:49.082 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1700 IOs, 734925824 bytes 2023-12-10T19:03:49.288 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1800 IOs, 785625088 bytes 2023-12-10T19:03:49.461 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1900 IOs, 826884096 bytes 2023-12-10T19:03:49.608 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2000 IOs, 854324224 bytes 2023-12-10T19:03:49.733 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2100 IOs, 892235776 bytes 2023-12-10T19:03:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2200 IOs, 957127680 bytes 2023-12-10T19:03:50.207 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2300 IOs, 1009866752 bytes 2023-12-10T19:03:50.471 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2400 IOs, 1087705088 bytes 2023-12-10T19:03:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2500 IOs, 1102100480 bytes 2023-12-10T19:03:50.761 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2600 IOs, 1146799104 bytes 2023-12-10T19:03:50.953 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2700 IOs, 1190513664 bytes 2023-12-10T19:03:51.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2800 IOs, 1275482112 bytes 2023-12-10T19:03:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2900 IOs, 1289499648 bytes 2023-12-10T19:03:51.553 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3000 IOs, 1331282944 bytes 2023-12-10T19:03:51.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3100 IOs, 1350538240 bytes 2023-12-10T19:03:51.901 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3200 IOs, 1411949568 bytes 2023-12-10T19:03:52.094 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3300 IOs, 1469280256 bytes 2023-12-10T19:03:52.309 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3400 IOs, 1521944576 bytes 2023-12-10T19:03:52.422 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3500 IOs, 1532359680 bytes 2023-12-10T19:03:52.531 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3600 IOs, 1543674880 bytes 2023-12-10T19:03:52.633 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3700 IOs, 1552229376 bytes 2023-12-10T19:03:52.777 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3800 IOs, 1582161920 bytes 2023-12-10T19:03:52.950 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3900 IOs, 1627387904 bytes 2023-12-10T19:03:53.167 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4000 IOs, 1694083072 bytes 2023-12-10T19:03:53.288 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4100 IOs, 1731738624 bytes 2023-12-10T19:03:53.424 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4200 IOs, 1732354048 bytes 2023-12-10T19:03:53.530 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4300 IOs, 1759551488 bytes 2023-12-10T19:03:53.719 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4400 IOs, 1791510528 bytes 2023-12-10T19:03:53.880 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4500 IOs, 1819616256 bytes 2023-12-10T19:03:54.087 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4600 IOs, 1879131136 bytes 2023-12-10T19:03:54.296 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4700 IOs, 1934351360 bytes 2023-12-10T19:03:54.387 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4800 IOs, 1934959616 bytes 2023-12-10T19:03:54.535 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4900 IOs, 1973498880 bytes 2023-12-10T19:03:54.693 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5000 IOs, 2001652736 bytes 2023-12-10T19:03:54.877 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5100 IOs, 2042047488 bytes 2023-12-10T19:03:55.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5200 IOs, 2109271040 bytes 2023-12-10T19:03:55.249 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5300 IOs, 2135655424 bytes 2023-12-10T19:03:55.389 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5400 IOs, 2185428992 bytes 2023-12-10T19:03:55.577 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5500 IOs, 2223348736 bytes 2023-12-10T19:03:55.731 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5600 IOs, 2234714112 bytes 2023-12-10T19:03:55.834 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5700 IOs, 2244293632 bytes 2023-12-10T19:03:56.024 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5800 IOs, 2298450944 bytes 2023-12-10T19:03:56.159 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5900 IOs, 2321031168 bytes 2023-12-10T19:03:56.312 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6000 IOs, 2364063744 bytes 2023-12-10T19:03:56.504 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6100 IOs, 2413617152 bytes 2023-12-10T19:03:56.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6200 IOs, 2481127424 bytes 2023-12-10T19:03:56.899 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6300 IOs, 2517556224 bytes 2023-12-10T19:03:57.062 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6400 IOs, 2556422144 bytes 2023-12-10T19:03:57.287 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6500 IOs, 2602964992 bytes 2023-12-10T19:03:57.406 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6600 IOs, 2618508288 bytes 2023-12-10T19:03:57.583 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6700 IOs, 2654538752 bytes 2023-12-10T19:03:57.782 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6800 IOs, 2702574592 bytes 2023-12-10T19:03:57.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=124 2023-12-10T19:03:57.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T19:03:57.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 124 -eq 124 ] 2023-12-10T19:03:57.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:03:57.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_8 2023-12-10T19:03:57.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image_and_enable_mirror cluster2 mirror image_8 snapshot 128M 2023-12-10T19:03:57.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:03:57.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:03:57.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:03:57.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:03:57.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_8 2023-12-10T19:03:57.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:03:57.978 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:03:57.978 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n snapshot ] 2023-12-10T19:03:57.978 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:03:57.978 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image cluster2 mirror image_8 128M 2023-12-10T19:03:57.978 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:03:57.978 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:03:57.978 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:03:57.978 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:03:57.978 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_8 2023-12-10T19:03:57.978 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:03:57.978 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local size=128 2023-12-10T19:03:57.978 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n 128M ] 2023-12-10T19:03:57.978 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ size=128M 2023-12-10T19:03:57.978 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:03:57.978 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_8 2023-12-10T19:03:58.025 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ is_pool_mirror_mode_image mirror 2023-12-10T19:03:58.026 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:03:58.026 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ image = image ] 2023-12-10T19:03:58.026 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:03:58.026 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ enable_mirror cluster2 mirror image_8 snapshot 2023-12-10T19:03:58.026 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:03:58.026 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:03:58.026 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_8 2023-12-10T19:03:58.026 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:03:58.026 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 snapshot 2023-12-10T19:03:58.463 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirroring enabled 2023-12-10T19:03:58.472 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 info mirror/image_8 2023-12-10T19:03:58.517 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_8': 2023-12-10T19:03:58.517 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:03:58.517 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:03:58.517 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 1 2023-12-10T19:03:58.518 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 12c2877f54d 2023-12-10T19:03:58.518 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.12c2877f54d 2023-12-10T19:03:58.518 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:03:58.518 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering 2023-12-10T19:03:58.518 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:03:58.518 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:03:58.518 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:03:58 2023 2023-12-10T19:03:58.518 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:03:58 2023 2023-12-10T19:03:58.518 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:03:58 2023 2023-12-10T19:03:58.518 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:03:58.518 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:03:58.518 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: f24c77de-98cd-45a4-9355-6e2e919e8517 2023-12-10T19:03:58.518 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: true 2023-12-10T19:03:58.521 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T19:03:58.522 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stress_write_image cluster2 mirror image_8 2023-12-10T19:03:58.522 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:03:58.522 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:03:58.522 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_8 2023-12-10T19:03:58.522 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-10T19:03:58.523 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local duration=11 2023-12-10T19:03:58.523 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T19:03:58.523 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2023-12-10T19:03:58.680 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:100 IOs, 9507840 bytes 2023-12-10T19:03:58.823 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:200 IOs, 37274624 bytes 2023-12-10T19:03:58.941 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:300 IOs, 65478656 bytes 2023-12-10T19:03:59.155 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:400 IOs, 129639424 bytes 2023-12-10T19:03:59.327 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:500 IOs, 177356800 bytes 2023-12-10T19:03:59.475 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:600 IOs, 199779328 bytes 2023-12-10T19:03:59.578 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:700 IOs, 209335296 bytes 2023-12-10T19:03:59.789 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:800 IOs, 278863872 bytes 2023-12-10T19:03:59.942 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:900 IOs, 317076480 bytes 2023-12-10T19:04:00.099 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1000 IOs, 350487552 bytes 2023-12-10T19:04:00.230 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1100 IOs, 372441088 bytes 2023-12-10T19:04:00.413 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1200 IOs, 420971520 bytes 2023-12-10T19:04:00.606 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1300 IOs, 475918336 bytes 2023-12-10T19:04:00.753 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1400 IOs, 497913856 bytes 2023-12-10T19:04:01.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1500 IOs, 635104256 bytes 2023-12-10T19:04:01.425 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1600 IOs, 710115328 bytes 2023-12-10T19:04:01.605 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1700 IOs, 747822080 bytes 2023-12-10T19:04:01.759 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1800 IOs, 779602944 bytes 2023-12-10T19:04:01.937 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1900 IOs, 824651776 bytes 2023-12-10T19:04:02.102 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2000 IOs, 887453696 bytes 2023-12-10T19:04:02.245 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2100 IOs, 908783616 bytes 2023-12-10T19:04:02.433 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2200 IOs, 947643392 bytes 2023-12-10T19:04:02.586 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2300 IOs, 977722368 bytes 2023-12-10T19:04:02.842 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2400 IOs, 1071453184 bytes 2023-12-10T19:04:03.099 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2500 IOs, 1125094400 bytes 2023-12-10T19:04:03.272 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2600 IOs, 1157936128 bytes 2023-12-10T19:04:03.441 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2700 IOs, 1187927040 bytes 2023-12-10T19:04:03.689 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2800 IOs, 1254572032 bytes 2023-12-10T19:04:03.892 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2900 IOs, 1298132992 bytes 2023-12-10T19:04:04.088 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3000 IOs, 1341827072 bytes 2023-12-10T19:04:04.282 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3100 IOs, 1388547072 bytes 2023-12-10T19:04:04.579 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3200 IOs, 1482645504 bytes 2023-12-10T19:04:04.684 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3300 IOs, 1484376064 bytes 2023-12-10T19:04:04.872 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3400 IOs, 1523724288 bytes 2023-12-10T19:04:04.996 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3500 IOs, 1539768320 bytes 2023-12-10T19:04:05.143 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3600 IOs, 1559205888 bytes 2023-12-10T19:04:05.298 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3700 IOs, 1589216256 bytes 2023-12-10T19:04:05.467 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3800 IOs, 1609262080 bytes 2023-12-10T19:04:05.699 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3900 IOs, 1667162112 bytes 2023-12-10T19:04:05.952 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4000 IOs, 1742136320 bytes 2023-12-10T19:04:06.071 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4100 IOs, 1753207808 bytes 2023-12-10T19:04:06.304 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4200 IOs, 1822262272 bytes 2023-12-10T19:04:06.402 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4300 IOs, 1822868480 bytes 2023-12-10T19:04:06.587 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4400 IOs, 1862556672 bytes 2023-12-10T19:04:06.755 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4500 IOs, 1896279040 bytes 2023-12-10T19:04:06.937 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4600 IOs, 1944310784 bytes 2023-12-10T19:04:07.169 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4700 IOs, 2006381568 bytes 2023-12-10T19:04:07.402 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4800 IOs, 2078160896 bytes 2023-12-10T19:04:07.524 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4900 IOs, 2078787584 bytes 2023-12-10T19:04:07.631 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5000 IOs, 2085390336 bytes 2023-12-10T19:04:07.766 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5100 IOs, 2098596864 bytes 2023-12-10T19:04:08.031 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5200 IOs, 2182985728 bytes 2023-12-10T19:04:08.170 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5300 IOs, 2205892608 bytes 2023-12-10T19:04:08.343 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5400 IOs, 2243389440 bytes 2023-12-10T19:04:08.451 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5500 IOs, 2247898112 bytes 2023-12-10T19:04:08.585 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5600 IOs, 2264875008 bytes 2023-12-10T19:04:08.714 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5700 IOs, 2285014016 bytes 2023-12-10T19:04:08.885 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5800 IOs, 2321054720 bytes 2023-12-10T19:04:09.072 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5900 IOs, 2370452480 bytes 2023-12-10T19:04:09.274 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6000 IOs, 2424356864 bytes 2023-12-10T19:04:09.404 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6100 IOs, 2443673600 bytes 2023-12-10T19:04:09.527 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=124 2023-12-10T19:04:09.527 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T19:04:09.527 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 124 -eq 124 ] 2023-12-10T19:04:09.527 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:04:09.527 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_9 2023-12-10T19:04:09.527 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image_and_enable_mirror cluster2 mirror image_9 snapshot 128M 2023-12-10T19:04:09.527 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:04:09.527 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:04:09.528 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:04:09.528 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:04:09.528 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_9 2023-12-10T19:04:09.528 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:04:09.528 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:04:09.528 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n snapshot ] 2023-12-10T19:04:09.528 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:04:09.528 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image cluster2 mirror image_9 128M 2023-12-10T19:04:09.528 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:04:09.528 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:04:09.528 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:04:09.528 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:04:09.528 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_9 2023-12-10T19:04:09.528 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:04:09.528 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local size=128 2023-12-10T19:04:09.529 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n 128M ] 2023-12-10T19:04:09.529 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ size=128M 2023-12-10T19:04:09.529 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:04:09.529 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_9 2023-12-10T19:04:09.574 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ is_pool_mirror_mode_image mirror 2023-12-10T19:04:09.574 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:04:09.574 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ image = image ] 2023-12-10T19:04:09.574 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:04:09.574 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ enable_mirror cluster2 mirror image_9 snapshot 2023-12-10T19:04:09.574 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:04:09.574 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:04:09.575 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_9 2023-12-10T19:04:09.575 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:04:09.575 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_9 snapshot 2023-12-10T19:04:10.473 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirroring enabled 2023-12-10T19:04:10.485 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 info mirror/image_9 2023-12-10T19:04:10.534 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_9': 2023-12-10T19:04:10.534 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:04:10.534 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:04:10.534 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 1 2023-12-10T19:04:10.534 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 12ce98e850cd 2023-12-10T19:04:10.534 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.12ce98e850cd 2023-12-10T19:04:10.534 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:04:10.534 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering 2023-12-10T19:04:10.534 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:04:10.534 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:04:10.534 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:04:09 2023 2023-12-10T19:04:10.534 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:04:09 2023 2023-12-10T19:04:10.534 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:04:09 2023 2023-12-10T19:04:10.535 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:04:10.535 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:04:10.535 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: ab1975bd-7976-4b27-85c2-606054317fe6 2023-12-10T19:04:10.535 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: true 2023-12-10T19:04:10.538 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T19:04:10.538 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stress_write_image cluster2 mirror image_9 2023-12-10T19:04:10.538 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:04:10.538 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:04:10.538 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_9 2023-12-10T19:04:10.538 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-10T19:04:10.541 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local duration=10 2023-12-10T19:04:10.541 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T19:04:10.541 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2023-12-10T19:04:10.788 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:100 IOs, 55678976 bytes 2023-12-10T19:04:11.030 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:200 IOs, 130821120 bytes 2023-12-10T19:04:11.200 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:300 IOs, 173027328 bytes 2023-12-10T19:04:11.353 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:400 IOs, 196962304 bytes 2023-12-10T19:04:11.493 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:500 IOs, 221396992 bytes 2023-12-10T19:04:11.671 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:600 IOs, 263948288 bytes 2023-12-10T19:04:11.863 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:700 IOs, 321127424 bytes 2023-12-10T19:04:12.016 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:800 IOs, 360243200 bytes 2023-12-10T19:04:12.260 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:900 IOs, 430967808 bytes 2023-12-10T19:04:12.456 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1000 IOs, 479619072 bytes 2023-12-10T19:04:12.634 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1100 IOs, 489729024 bytes 2023-12-10T19:04:12.868 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1200 IOs, 564898816 bytes 2023-12-10T19:04:13.049 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1300 IOs, 602730496 bytes 2023-12-10T19:04:13.246 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1400 IOs, 660117504 bytes 2023-12-10T19:04:13.514 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1500 IOs, 750702592 bytes 2023-12-10T19:04:13.627 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1600 IOs, 765679616 bytes 2023-12-10T19:04:13.748 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1700 IOs, 783517696 bytes 2023-12-10T19:04:13.863 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1800 IOs, 797577216 bytes 2023-12-10T19:04:14.042 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1900 IOs, 844284928 bytes 2023-12-10T19:04:14.257 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2000 IOs, 907409408 bytes 2023-12-10T19:04:14.482 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2100 IOs, 976954368 bytes 2023-12-10T19:04:14.637 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2200 IOs, 1006104576 bytes 2023-12-10T19:04:14.807 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2300 IOs, 1046154240 bytes 2023-12-10T19:04:14.922 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2400 IOs, 1057335296 bytes 2023-12-10T19:04:15.247 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2500 IOs, 1175869440 bytes 2023-12-10T19:04:15.397 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2600 IOs, 1206917120 bytes 2023-12-10T19:04:15.548 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2700 IOs, 1229663232 bytes 2023-12-10T19:04:15.640 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2800 IOs, 1230312448 bytes 2023-12-10T19:04:15.858 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2900 IOs, 1311949824 bytes 2023-12-10T19:04:16.101 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3000 IOs, 1389580288 bytes 2023-12-10T19:04:16.221 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3100 IOs, 1410880512 bytes 2023-12-10T19:04:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3200 IOs, 1423291392 bytes 2023-12-10T19:04:16.498 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3300 IOs, 1465537536 bytes 2023-12-10T19:04:16.694 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3400 IOs, 1515495424 bytes 2023-12-10T19:04:16.793 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3500 IOs, 1516134400 bytes 2023-12-10T19:04:16.956 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3600 IOs, 1553592320 bytes 2023-12-10T19:04:17.111 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3700 IOs, 1593521152 bytes 2023-12-10T19:04:17.279 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3800 IOs, 1618511872 bytes 2023-12-10T19:04:17.540 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3900 IOs, 1697033216 bytes 2023-12-10T19:04:17.677 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4000 IOs, 1706708992 bytes 2023-12-10T19:04:17.829 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4100 IOs, 1735076864 bytes 2023-12-10T19:04:17.989 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4200 IOs, 1772909568 bytes 2023-12-10T19:04:18.119 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4300 IOs, 1791323136 bytes 2023-12-10T19:04:18.383 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4400 IOs, 1877937152 bytes 2023-12-10T19:04:18.573 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4500 IOs, 1934168064 bytes 2023-12-10T19:04:18.716 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4600 IOs, 1958970368 bytes 2023-12-10T19:04:18.949 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4700 IOs, 2048667648 bytes 2023-12-10T19:04:19.149 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4800 IOs, 2096820224 bytes 2023-12-10T19:04:19.288 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4900 IOs, 2112623616 bytes 2023-12-10T19:04:19.458 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5000 IOs, 2186066944 bytes 2023-12-10T19:04:19.635 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5100 IOs, 2186677248 bytes 2023-12-10T19:04:19.822 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5200 IOs, 2229983232 bytes 2023-12-10T19:04:20.104 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5300 IOs, 2329017344 bytes 2023-12-10T19:04:20.224 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5400 IOs, 2336303104 bytes 2023-12-10T19:04:20.415 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5500 IOs, 2378300416 bytes 2023-12-10T19:04:20.546 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=124 2023-12-10T19:04:20.546 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T19:04:20.546 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 124 -eq 124 ] 2023-12-10T19:04:20.546 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:04:20.546 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_10 2023-12-10T19:04:20.546 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image_and_enable_mirror cluster2 mirror image_10 snapshot 128M 2023-12-10T19:04:20.546 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:04:20.546 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:04:20.546 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:04:20.546 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:04:20.546 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_10 2023-12-10T19:04:20.546 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:04:20.546 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:04:20.547 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n snapshot ] 2023-12-10T19:04:20.547 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:04:20.547 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image cluster2 mirror image_10 128M 2023-12-10T19:04:20.547 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:04:20.547 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:04:20.547 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:04:20.547 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:04:20.547 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_10 2023-12-10T19:04:20.547 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:04:20.547 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local size=128 2023-12-10T19:04:20.547 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n 128M ] 2023-12-10T19:04:20.547 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ size=128M 2023-12-10T19:04:20.547 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:04:20.547 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_10 2023-12-10T19:04:20.589 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ is_pool_mirror_mode_image mirror 2023-12-10T19:04:20.590 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:04:20.590 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ image = image ] 2023-12-10T19:04:20.590 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:04:20.590 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ enable_mirror cluster2 mirror image_10 snapshot 2023-12-10T19:04:20.590 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:04:20.590 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:04:20.590 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_10 2023-12-10T19:04:20.590 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:04:20.590 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_10 snapshot 2023-12-10T19:04:21.474 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirroring enabled 2023-12-10T19:04:21.486 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 info mirror/image_10 2023-12-10T19:04:21.532 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_10': 2023-12-10T19:04:21.532 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:04:21.532 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:04:21.532 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 1 2023-12-10T19:04:21.533 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 12dadac2f792 2023-12-10T19:04:21.533 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.12dadac2f792 2023-12-10T19:04:21.533 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:04:21.533 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering 2023-12-10T19:04:21.533 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:04:21.533 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:04:21.533 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:04:20 2023 2023-12-10T19:04:21.533 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:04:20 2023 2023-12-10T19:04:21.533 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:04:20 2023 2023-12-10T19:04:21.533 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:04:21.533 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:04:21.533 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: 5e79b28d-acd2-4fbc-9a84-6c7b4c7dc476 2023-12-10T19:04:21.533 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: true 2023-12-10T19:04:21.537 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T19:04:21.537 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stress_write_image cluster2 mirror image_10 2023-12-10T19:04:21.537 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:04:21.537 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:04:21.537 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_10 2023-12-10T19:04:21.537 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-10T19:04:21.538 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local duration=6 2023-12-10T19:04:21.538 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T19:04:21.538 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2023-12-10T19:04:21.806 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:100 IOs, 64935936 bytes 2023-12-10T19:04:21.916 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:200 IOs, 78695424 bytes 2023-12-10T19:04:22.141 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:300 IOs, 147077120 bytes 2023-12-10T19:04:22.414 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:400 IOs, 239208448 bytes 2023-12-10T19:04:22.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:500 IOs, 315830272 bytes 2023-12-10T19:04:22.815 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:600 IOs, 346885120 bytes 2023-12-10T19:04:22.909 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:700 IOs, 347500544 bytes 2023-12-10T19:04:23.082 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:800 IOs, 394834944 bytes 2023-12-10T19:04:23.296 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:900 IOs, 455126016 bytes 2023-12-10T19:04:23.558 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1000 IOs, 536287232 bytes 2023-12-10T19:04:23.693 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1100 IOs, 558392320 bytes 2023-12-10T19:04:23.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1200 IOs, 575037440 bytes 2023-12-10T19:04:24.017 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1300 IOs, 648783872 bytes 2023-12-10T19:04:24.236 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1400 IOs, 693620736 bytes 2023-12-10T19:04:24.377 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1500 IOs, 741955584 bytes 2023-12-10T19:04:24.714 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1600 IOs, 846329856 bytes 2023-12-10T19:04:24.878 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1700 IOs, 905338880 bytes 2023-12-10T19:04:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1800 IOs, 936536064 bytes 2023-12-10T19:04:25.204 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1900 IOs, 947369984 bytes 2023-12-10T19:04:25.362 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2000 IOs, 998247424 bytes 2023-12-10T19:04:25.543 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2100 IOs, 1045369856 bytes 2023-12-10T19:04:25.841 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2200 IOs, 1133608960 bytes 2023-12-10T19:04:25.956 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2300 IOs, 1150430208 bytes 2023-12-10T19:04:26.161 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2400 IOs, 1201924096 bytes 2023-12-10T19:04:26.477 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2500 IOs, 1308115968 bytes 2023-12-10T19:04:26.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2600 IOs, 1381451776 bytes 2023-12-10T19:04:26.858 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2700 IOs, 1408878592 bytes 2023-12-10T19:04:26.991 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2800 IOs, 1425961984 bytes 2023-12-10T19:04:27.217 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2900 IOs, 1492353024 bytes 2023-12-10T19:04:27.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3000 IOs, 1510571008 bytes 2023-12-10T19:04:27.543 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=124 2023-12-10T19:04:27.543 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T19:04:27.543 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 124 -eq 124 ] 2023-12-10T19:04:27.543 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:04:27.544 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_11 2023-12-10T19:04:27.544 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image_and_enable_mirror cluster2 mirror image_11 snapshot 128M 2023-12-10T19:04:27.544 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:04:27.544 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:04:27.544 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:04:27.544 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:04:27.544 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_11 2023-12-10T19:04:27.544 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:04:27.544 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:04:27.544 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n snapshot ] 2023-12-10T19:04:27.544 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:04:27.544 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image cluster2 mirror image_11 128M 2023-12-10T19:04:27.544 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:04:27.544 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:04:27.544 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:04:27.545 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:04:27.545 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_11 2023-12-10T19:04:27.545 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:04:27.545 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local size=128 2023-12-10T19:04:27.545 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n 128M ] 2023-12-10T19:04:27.545 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ size=128M 2023-12-10T19:04:27.545 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:04:27.545 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_11 2023-12-10T19:04:27.591 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ is_pool_mirror_mode_image mirror 2023-12-10T19:04:27.591 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:04:27.591 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ image = image ] 2023-12-10T19:04:27.591 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:04:27.591 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ enable_mirror cluster2 mirror image_11 snapshot 2023-12-10T19:04:27.591 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:04:27.591 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:04:27.591 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_11 2023-12-10T19:04:27.591 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:04:27.591 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_11 snapshot 2023-12-10T19:04:28.475 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirroring enabled 2023-12-10T19:04:28.487 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 info mirror/image_11 2023-12-10T19:04:28.533 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_11': 2023-12-10T19:04:28.533 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:04:28.533 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:04:28.533 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 1 2023-12-10T19:04:28.533 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 12e62ad686f7 2023-12-10T19:04:28.533 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.12e62ad686f7 2023-12-10T19:04:28.533 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:04:28.533 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering 2023-12-10T19:04:28.534 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:04:28.534 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:04:28.534 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:04:27 2023 2023-12-10T19:04:28.534 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:04:27 2023 2023-12-10T19:04:28.534 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:04:27 2023 2023-12-10T19:04:28.534 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:04:28.534 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:04:28.534 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: d372b711-4545-4edb-a533-b051b75e68e3 2023-12-10T19:04:28.534 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: true 2023-12-10T19:04:28.537 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T19:04:28.537 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stress_write_image cluster2 mirror image_11 2023-12-10T19:04:28.537 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:04:28.537 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:04:28.537 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_11 2023-12-10T19:04:28.538 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-10T19:04:28.538 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local duration=10 2023-12-10T19:04:28.539 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T19:04:28.539 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_11 --debug-rbd=20 --debug-journaler=20 2023-12-10T19:04:28.751 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:100 IOs, 32983040 bytes 2023-12-10T19:04:28.941 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:200 IOs, 77228032 bytes 2023-12-10T19:04:29.142 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:300 IOs, 128278528 bytes 2023-12-10T19:04:29.387 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:400 IOs, 198541312 bytes 2023-12-10T19:04:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:500 IOs, 223697920 bytes 2023-12-10T19:04:29.713 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:600 IOs, 272612352 bytes 2023-12-10T19:04:29.852 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:700 IOs, 305034240 bytes 2023-12-10T19:04:30.088 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:800 IOs, 383972352 bytes 2023-12-10T19:04:30.239 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:900 IOs, 411721728 bytes 2023-12-10T19:04:30.351 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1000 IOs, 449523712 bytes 2023-12-10T19:04:30.595 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1100 IOs, 492532736 bytes 2023-12-10T19:04:30.808 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1200 IOs, 554296320 bytes 2023-12-10T19:04:31.017 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1300 IOs, 607029248 bytes 2023-12-10T19:04:31.151 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1400 IOs, 636080128 bytes 2023-12-10T19:04:31.415 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1500 IOs, 718586880 bytes 2023-12-10T19:04:31.622 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1600 IOs, 773000192 bytes 2023-12-10T19:04:31.825 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1700 IOs, 843680768 bytes 2023-12-10T19:04:32.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1800 IOs, 929760256 bytes 2023-12-10T19:04:32.228 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1900 IOs, 958330880 bytes 2023-12-10T19:04:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2000 IOs, 979627008 bytes 2023-12-10T19:04:32.542 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2100 IOs, 1025895424 bytes 2023-12-10T19:04:32.731 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2200 IOs, 1072219136 bytes 2023-12-10T19:04:32.902 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2300 IOs, 1111805952 bytes 2023-12-10T19:04:33.020 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2400 IOs, 1125027840 bytes 2023-12-10T19:04:33.197 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2500 IOs, 1171141632 bytes 2023-12-10T19:04:33.345 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2600 IOs, 1195939840 bytes 2023-12-10T19:04:33.502 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2700 IOs, 1232247808 bytes 2023-12-10T19:04:33.611 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2800 IOs, 1237872640 bytes 2023-12-10T19:04:33.813 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2900 IOs, 1291488256 bytes 2023-12-10T19:04:33.955 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3000 IOs, 1315080192 bytes 2023-12-10T19:04:34.084 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3100 IOs, 1356469248 bytes 2023-12-10T19:04:34.420 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3200 IOs, 1462249472 bytes 2023-12-10T19:04:34.572 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3300 IOs, 1490298880 bytes 2023-12-10T19:04:34.854 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3400 IOs, 1577865216 bytes 2023-12-10T19:04:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3500 IOs, 1601894400 bytes 2023-12-10T19:04:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3600 IOs, 1602513920 bytes 2023-12-10T19:04:35.310 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3700 IOs, 1651231744 bytes 2023-12-10T19:04:35.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3800 IOs, 1719485440 bytes 2023-12-10T19:04:35.757 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3900 IOs, 1770932224 bytes 2023-12-10T19:04:35.944 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4000 IOs, 1812764672 bytes 2023-12-10T19:04:36.207 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4100 IOs, 1897259008 bytes 2023-12-10T19:04:36.386 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4200 IOs, 1930697728 bytes 2023-12-10T19:04:36.496 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4300 IOs, 1934590976 bytes 2023-12-10T19:04:36.759 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4400 IOs, 1988166656 bytes 2023-12-10T19:04:36.921 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4500 IOs, 2017442816 bytes 2023-12-10T19:04:37.064 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4600 IOs, 2039770112 bytes 2023-12-10T19:04:37.261 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4700 IOs, 2086280192 bytes 2023-12-10T19:04:37.368 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4800 IOs, 2089345024 bytes 2023-12-10T19:04:37.653 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4900 IOs, 2174471168 bytes 2023-12-10T19:04:37.836 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5000 IOs, 2214646784 bytes 2023-12-10T19:04:38.035 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5100 IOs, 2285704192 bytes 2023-12-10T19:04:38.287 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5200 IOs, 2335234048 bytes 2023-12-10T19:04:38.482 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5300 IOs, 2377154560 bytes 2023-12-10T19:04:38.543 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=124 2023-12-10T19:04:38.543 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T19:04:38.543 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 124 -eq 124 ] 2023-12-10T19:04:38.543 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:04:38.543 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_12 2023-12-10T19:04:38.543 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image_and_enable_mirror cluster2 mirror image_12 snapshot 128M 2023-12-10T19:04:38.543 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:04:38.543 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:04:38.543 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:04:38.544 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:04:38.544 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_12 2023-12-10T19:04:38.544 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:04:38.544 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:04:38.544 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n snapshot ] 2023-12-10T19:04:38.544 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:04:38.544 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image cluster2 mirror image_12 128M 2023-12-10T19:04:38.544 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:04:38.544 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:04:38.544 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:04:38.544 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:04:38.544 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_12 2023-12-10T19:04:38.544 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:04:38.544 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local size=128 2023-12-10T19:04:38.544 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n 128M ] 2023-12-10T19:04:38.545 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ size=128M 2023-12-10T19:04:38.545 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:04:38.545 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_12 2023-12-10T19:04:38.592 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ is_pool_mirror_mode_image mirror 2023-12-10T19:04:38.592 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:04:38.592 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ image = image ] 2023-12-10T19:04:38.592 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:04:38.593 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ enable_mirror cluster2 mirror image_12 snapshot 2023-12-10T19:04:38.593 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:04:38.593 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:04:38.593 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_12 2023-12-10T19:04:38.593 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:04:38.593 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_12 snapshot 2023-12-10T19:04:39.485 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirroring enabled 2023-12-10T19:04:39.496 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 info mirror/image_12 2023-12-10T19:04:39.543 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_12': 2023-12-10T19:04:39.543 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:04:39.544 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:04:39.544 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 1 2023-12-10T19:04:39.544 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 12f2a85fa5a 2023-12-10T19:04:39.544 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.12f2a85fa5a 2023-12-10T19:04:39.544 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:04:39.544 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering 2023-12-10T19:04:39.544 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:04:39.544 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:04:39.544 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:04:38 2023 2023-12-10T19:04:39.544 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:04:38 2023 2023-12-10T19:04:39.544 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:04:38 2023 2023-12-10T19:04:39.544 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:04:39.544 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:04:39.544 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: f2b859e8-6dbd-4781-91cf-11df1eb8a7a0 2023-12-10T19:04:39.544 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: true 2023-12-10T19:04:39.548 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T19:04:39.548 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stress_write_image cluster2 mirror image_12 2023-12-10T19:04:39.548 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:04:39.548 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:04:39.548 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_12 2023-12-10T19:04:39.548 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-10T19:04:39.550 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local duration=11 2023-12-10T19:04:39.550 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T19:04:39.550 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_12 --debug-rbd=20 --debug-journaler=20 2023-12-10T19:04:39.761 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:100 IOs, 34738176 bytes 2023-12-10T19:04:39.941 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:200 IOs, 66190336 bytes 2023-12-10T19:04:40.111 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:300 IOs, 121009152 bytes 2023-12-10T19:04:40.353 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:400 IOs, 175709184 bytes 2023-12-10T19:04:40.554 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:500 IOs, 225943552 bytes 2023-12-10T19:04:40.681 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:600 IOs, 242112512 bytes 2023-12-10T19:04:40.924 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:700 IOs, 311651328 bytes 2023-12-10T19:04:41.109 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:800 IOs, 352959488 bytes 2023-12-10T19:04:41.255 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:900 IOs, 374125568 bytes 2023-12-10T19:04:41.442 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1000 IOs, 416843776 bytes 2023-12-10T19:04:41.719 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1100 IOs, 507244544 bytes 2023-12-10T19:04:41.859 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1200 IOs, 525639680 bytes 2023-12-10T19:04:42.012 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1300 IOs, 573140992 bytes 2023-12-10T19:04:42.305 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1400 IOs, 662764544 bytes 2023-12-10T19:04:42.586 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1500 IOs, 750041088 bytes 2023-12-10T19:04:42.788 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1600 IOs, 809715712 bytes 2023-12-10T19:04:42.972 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1700 IOs, 863561728 bytes 2023-12-10T19:04:43.196 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1800 IOs, 929973248 bytes 2023-12-10T19:04:43.543 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1900 IOs, 1031106560 bytes 2023-12-10T19:04:43.733 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2000 IOs, 1103341568 bytes 2023-12-10T19:04:44.062 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2100 IOs, 1167884288 bytes 2023-12-10T19:04:44.368 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2200 IOs, 1275572224 bytes 2023-12-10T19:04:44.511 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2300 IOs, 1303952384 bytes 2023-12-10T19:04:44.690 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2400 IOs, 1353589760 bytes 2023-12-10T19:04:44.809 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2500 IOs, 1366166528 bytes 2023-12-10T19:04:45.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2600 IOs, 1433783296 bytes 2023-12-10T19:04:45.176 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2700 IOs, 1452304384 bytes 2023-12-10T19:04:45.350 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2800 IOs, 1482959872 bytes 2023-12-10T19:04:45.609 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2900 IOs, 1560039424 bytes 2023-12-10T19:04:45.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3000 IOs, 1611919360 bytes 2023-12-10T19:04:45.960 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3100 IOs, 1633065984 bytes 2023-12-10T19:04:46.182 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3200 IOs, 1703365632 bytes 2023-12-10T19:04:46.296 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3300 IOs, 1703933952 bytes 2023-12-10T19:04:46.448 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3400 IOs, 1727596544 bytes 2023-12-10T19:04:46.701 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3500 IOs, 1798704128 bytes 2023-12-10T19:04:46.912 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3600 IOs, 1874924544 bytes 2023-12-10T19:04:47.197 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3700 IOs, 1965785088 bytes 2023-12-10T19:04:47.433 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3800 IOs, 2025926656 bytes 2023-12-10T19:04:47.593 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3900 IOs, 2051924992 bytes 2023-12-10T19:04:47.729 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4000 IOs, 2065981440 bytes 2023-12-10T19:04:47.856 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4100 IOs, 2082189312 bytes 2023-12-10T19:04:48.036 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4200 IOs, 2130889728 bytes 2023-12-10T19:04:48.184 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4300 IOs, 2153587712 bytes 2023-12-10T19:04:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4400 IOs, 2201103360 bytes 2023-12-10T19:04:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4500 IOs, 2228469760 bytes 2023-12-10T19:04:48.630 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4600 IOs, 2238680064 bytes 2023-12-10T19:04:48.862 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4700 IOs, 2307046400 bytes 2023-12-10T19:04:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4800 IOs, 2332299264 bytes 2023-12-10T19:04:49.169 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4900 IOs, 2383195136 bytes 2023-12-10T19:04:49.352 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5000 IOs, 2414415872 bytes 2023-12-10T19:04:49.449 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5100 IOs, 2440400896 bytes 2023-12-10T19:04:49.685 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5200 IOs, 2485356544 bytes 2023-12-10T19:04:49.889 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5300 IOs, 2550069248 bytes 2023-12-10T19:04:50.152 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5400 IOs, 2602786816 bytes 2023-12-10T19:04:50.479 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5500 IOs, 2724344832 bytes 2023-12-10T19:04:50.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=124 2023-12-10T19:04:50.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T19:04:50.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 124 -eq 124 ] 2023-12-10T19:04:50.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:04:50.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_13 2023-12-10T19:04:50.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image_and_enable_mirror cluster2 mirror image_13 snapshot 128M 2023-12-10T19:04:50.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:04:50.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:04:50.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:04:50.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:04:50.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_13 2023-12-10T19:04:50.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:04:50.558 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:04:50.558 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n snapshot ] 2023-12-10T19:04:50.558 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:04:50.558 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image cluster2 mirror image_13 128M 2023-12-10T19:04:50.558 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:04:50.558 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:04:50.558 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:04:50.558 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:04:50.558 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_13 2023-12-10T19:04:50.558 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:04:50.558 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local size=128 2023-12-10T19:04:50.558 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n 128M ] 2023-12-10T19:04:50.558 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ size=128M 2023-12-10T19:04:50.558 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:04:50.558 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_13 2023-12-10T19:04:50.604 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ is_pool_mirror_mode_image mirror 2023-12-10T19:04:50.605 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:04:50.605 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ image = image ] 2023-12-10T19:04:50.605 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:04:50.605 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ enable_mirror cluster2 mirror image_13 snapshot 2023-12-10T19:04:50.605 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:04:50.605 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:04:50.605 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_13 2023-12-10T19:04:50.605 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:04:50.605 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_13 snapshot 2023-12-10T19:04:51.474 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirroring enabled 2023-12-10T19:04:51.483 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 info mirror/image_13 2023-12-10T19:04:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_13': 2023-12-10T19:04:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:04:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:04:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 1 2023-12-10T19:04:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 12fe551e9aba 2023-12-10T19:04:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.12fe551e9aba 2023-12-10T19:04:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:04:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering 2023-12-10T19:04:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:04:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:04:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:04:50 2023 2023-12-10T19:04:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:04:50 2023 2023-12-10T19:04:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:04:50 2023 2023-12-10T19:04:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:04:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:04:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: 9cd9cf4d-71e8-4901-8340-6c85f5a313e8 2023-12-10T19:04:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: true 2023-12-10T19:04:51.743 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T19:04:51.743 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stress_write_image cluster2 mirror image_13 2023-12-10T19:04:51.743 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:04:51.743 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:04:51.743 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_13 2023-12-10T19:04:51.743 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-10T19:04:51.745 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local duration=10 2023-12-10T19:04:51.745 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T19:04:51.745 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_13 --debug-rbd=20 --debug-journaler=20 2023-12-10T19:04:52.044 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:100 IOs, 80736256 bytes 2023-12-10T19:04:52.259 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:200 IOs, 149055488 bytes 2023-12-10T19:04:52.453 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:300 IOs, 209296384 bytes 2023-12-10T19:04:52.679 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:400 IOs, 284429312 bytes 2023-12-10T19:04:53.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:500 IOs, 434216960 bytes 2023-12-10T19:04:53.293 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:600 IOs, 465588224 bytes 2023-12-10T19:04:53.506 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:700 IOs, 518999040 bytes 2023-12-10T19:04:53.673 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:800 IOs, 554645504 bytes 2023-12-10T19:04:53.841 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:900 IOs, 585583616 bytes 2023-12-10T19:04:53.965 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1000 IOs, 619125760 bytes 2023-12-10T19:04:54.238 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1100 IOs, 701432832 bytes 2023-12-10T19:04:54.363 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1200 IOs, 707323904 bytes 2023-12-10T19:04:54.551 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1300 IOs, 771304448 bytes 2023-12-10T19:04:54.796 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1400 IOs, 825118720 bytes 2023-12-10T19:04:54.978 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1500 IOs, 888626176 bytes 2023-12-10T19:04:55.230 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1600 IOs, 939718656 bytes 2023-12-10T19:04:55.456 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1700 IOs, 999322624 bytes 2023-12-10T19:04:55.785 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1800 IOs, 1120177152 bytes 2023-12-10T19:04:55.978 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1900 IOs, 1185331200 bytes 2023-12-10T19:04:56.207 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2000 IOs, 1228880896 bytes 2023-12-10T19:04:56.397 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2100 IOs, 1270603776 bytes 2023-12-10T19:04:56.662 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2200 IOs, 1344512000 bytes 2023-12-10T19:04:56.831 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2300 IOs, 1380251648 bytes 2023-12-10T19:04:56.945 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2400 IOs, 1390286848 bytes 2023-12-10T19:04:57.239 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2500 IOs, 1489262592 bytes 2023-12-10T19:04:57.386 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2600 IOs, 1515597824 bytes 2023-12-10T19:04:57.489 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2700 IOs, 1516193792 bytes 2023-12-10T19:04:57.651 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2800 IOs, 1545536512 bytes 2023-12-10T19:04:57.835 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2900 IOs, 1594289152 bytes 2023-12-10T19:04:58.185 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3000 IOs, 1700934656 bytes 2023-12-10T19:04:58.346 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3100 IOs, 1734074368 bytes 2023-12-10T19:04:58.549 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3200 IOs, 1809125376 bytes 2023-12-10T19:04:58.766 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3300 IOs, 1863313408 bytes 2023-12-10T19:04:59.070 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3400 IOs, 1932811264 bytes 2023-12-10T19:04:59.283 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3500 IOs, 1982541824 bytes 2023-12-10T19:04:59.492 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3600 IOs, 2037718016 bytes 2023-12-10T19:04:59.611 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3700 IOs, 2052502528 bytes 2023-12-10T19:04:59.850 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3800 IOs, 2139761664 bytes 2023-12-10T19:04:59.987 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3900 IOs, 2147359744 bytes 2023-12-10T19:05:00.082 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4000 IOs, 2147974144 bytes 2023-12-10T19:05:00.253 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4100 IOs, 2186297344 bytes 2023-12-10T19:05:00.473 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4200 IOs, 2250062848 bytes 2023-12-10T19:05:00.622 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4300 IOs, 2279947264 bytes 2023-12-10T19:05:00.731 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5024823882890865) > weight (0.25) 2023-12-10T19:05:00.731 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2023-12-10T19:05:00.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2023-12-10T19:05:00.731 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2023-12-10T19:05:00.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi076.stderr:2023-12-10T19:05:00.738+0000 7fc5890f6640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 43841) UID: 1000 2023-12-10T19:05:00.781 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4400 IOs, 2317408256 bytes 2023-12-10T19:05:01.097 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4500 IOs, 2396649472 bytes 2023-12-10T19:05:01.270 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4600 IOs, 2434242560 bytes 2023-12-10T19:05:01.502 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4700 IOs, 2489430016 bytes 2023-12-10T19:05:01.693 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4800 IOs, 2531302400 bytes 2023-12-10T19:05:01.749 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=124 2023-12-10T19:05:01.749 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T19:05:01.749 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 124 -eq 124 ] 2023-12-10T19:05:01.750 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:05:01.750 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_14 2023-12-10T19:05:01.750 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image_and_enable_mirror cluster2 mirror image_14 snapshot 128M 2023-12-10T19:05:01.750 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:05:01.750 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:05:01.750 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:05:01.750 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:05:01.750 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_14 2023-12-10T19:05:01.750 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:05:01.750 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:05:01.750 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n snapshot ] 2023-12-10T19:05:01.750 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:05:01.750 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image cluster2 mirror image_14 128M 2023-12-10T19:05:01.750 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:05:01.751 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:05:01.751 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:05:01.751 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:05:01.751 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_14 2023-12-10T19:05:01.751 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:05:01.751 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local size=128 2023-12-10T19:05:01.751 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n 128M ] 2023-12-10T19:05:01.751 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ size=128M 2023-12-10T19:05:01.751 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:05:01.751 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_14 2023-12-10T19:05:01.801 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ is_pool_mirror_mode_image mirror 2023-12-10T19:05:01.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:05:01.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ image = image ] 2023-12-10T19:05:01.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:05:01.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ enable_mirror cluster2 mirror image_14 snapshot 2023-12-10T19:05:01.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:05:01.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:05:01.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_14 2023-12-10T19:05:01.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:05:01.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_14 snapshot 2023-12-10T19:05:02.477 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirroring enabled 2023-12-10T19:05:02.489 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 info mirror/image_14 2023-12-10T19:05:02.539 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_14': 2023-12-10T19:05:02.539 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:05:02.539 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:05:02.539 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 1 2023-12-10T19:05:02.539 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 1309691088f7 2023-12-10T19:05:02.540 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.1309691088f7 2023-12-10T19:05:02.540 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:05:02.540 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering 2023-12-10T19:05:02.540 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:05:02.540 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:05:02.540 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:05:01 2023 2023-12-10T19:05:02.540 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:05:01 2023 2023-12-10T19:05:02.540 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:05:01 2023 2023-12-10T19:05:02.540 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:05:02.540 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:05:02.540 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: 1d54e641-de62-4511-9abb-01dd3218783c 2023-12-10T19:05:02.540 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: true 2023-12-10T19:05:02.543 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T19:05:02.544 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stress_write_image cluster2 mirror image_14 2023-12-10T19:05:02.544 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:05:02.544 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:05:02.544 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_14 2023-12-10T19:05:02.544 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-10T19:05:02.545 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local duration=10 2023-12-10T19:05:02.545 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T19:05:02.545 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_14 --debug-rbd=20 --debug-journaler=20 2023-12-10T19:05:02.747 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:100 IOs, 33555456 bytes 2023-12-10T19:05:02.941 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:200 IOs, 80415744 bytes 2023-12-10T19:05:03.227 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:300 IOs, 179760128 bytes 2023-12-10T19:05:03.467 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:400 IOs, 252997632 bytes 2023-12-10T19:05:03.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:500 IOs, 308067328 bytes 2023-12-10T19:05:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:600 IOs, 350672896 bytes 2023-12-10T19:05:04.026 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:700 IOs, 394823680 bytes 2023-12-10T19:05:04.225 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:800 IOs, 463047680 bytes 2023-12-10T19:05:04.402 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:900 IOs, 478931968 bytes 2023-12-10T19:05:04.612 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1000 IOs, 529835008 bytes 2023-12-10T19:05:04.808 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1100 IOs, 575483904 bytes 2023-12-10T19:05:04.973 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1200 IOs, 613519360 bytes 2023-12-10T19:05:05.136 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1300 IOs, 656438272 bytes 2023-12-10T19:05:05.475 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1400 IOs, 767585280 bytes 2023-12-10T19:05:05.693 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1500 IOs, 827987968 bytes 2023-12-10T19:05:05.732 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2023-12-10T19:05:05.732 INFO:teuthology.orchestra.run:waiting for 600 2023-12-10T19:05:05.733 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2023-12-10T19:05:05.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-12-10T19:05:05.733 DEBUG:teuthology.orchestra.run.smithi076:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-12-10T19:05:05.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-12-10T19:05:05.736 INFO:tasks.rbd_mirror_thrash:waiting for 80 secs before thrashing 2023-12-10T19:05:05.909 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1600 IOs, 881283072 bytes 2023-12-10T19:05:06.068 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1700 IOs, 907956224 bytes 2023-12-10T19:05:06.244 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1800 IOs, 956111872 bytes 2023-12-10T19:05:06.448 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1900 IOs, 1003343872 bytes 2023-12-10T19:05:06.667 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2000 IOs, 1085629440 bytes 2023-12-10T19:05:06.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2100 IOs, 1095974912 bytes 2023-12-10T19:05:07.049 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2200 IOs, 1157071872 bytes 2023-12-10T19:05:07.186 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2300 IOs, 1202146304 bytes 2023-12-10T19:05:07.384 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2400 IOs, 1258978304 bytes 2023-12-10T19:05:07.595 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2500 IOs, 1288013824 bytes 2023-12-10T19:05:07.773 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2600 IOs, 1329165312 bytes 2023-12-10T19:05:08.007 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2700 IOs, 1388673024 bytes 2023-12-10T19:05:08.216 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2800 IOs, 1453864960 bytes 2023-12-10T19:05:08.380 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2900 IOs, 1506325504 bytes 2023-12-10T19:05:08.596 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3000 IOs, 1551938560 bytes 2023-12-10T19:05:08.731 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.39142903553913433) > weight (0.25) 2023-12-10T19:05:08.731 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.7409395628681953) > weight (0.25) 2023-12-10T19:05:08.731 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2023-12-10T19:05:08.732 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2023-12-10T19:05:08.732 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2023-12-10T19:05:08.733 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi076.stderr:2023-12-10T19:05:08.738+0000 7f34cd059640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 46512) UID: 1000 2023-12-10T19:05:08.790 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3100 IOs, 1597697024 bytes 2023-12-10T19:05:09.058 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3200 IOs, 1677566976 bytes 2023-12-10T19:05:09.213 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3300 IOs, 1711268864 bytes 2023-12-10T19:05:09.322 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3400 IOs, 1711879168 bytes 2023-12-10T19:05:09.442 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3500 IOs, 1728329728 bytes 2023-12-10T19:05:09.543 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3600 IOs, 1732928512 bytes 2023-12-10T19:05:09.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3700 IOs, 1758656512 bytes 2023-12-10T19:05:09.885 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3800 IOs, 1806332928 bytes 2023-12-10T19:05:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3900 IOs, 1880937472 bytes 2023-12-10T19:05:10.274 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4000 IOs, 1897989120 bytes 2023-12-10T19:05:10.467 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4100 IOs, 1947991040 bytes 2023-12-10T19:05:10.607 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4200 IOs, 1968310272 bytes 2023-12-10T19:05:10.806 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4300 IOs, 2014036992 bytes 2023-12-10T19:05:10.951 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4400 IOs, 2035766272 bytes 2023-12-10T19:05:11.178 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4500 IOs, 2092183552 bytes 2023-12-10T19:05:11.385 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4600 IOs, 2141411328 bytes 2023-12-10T19:05:11.559 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4700 IOs, 2188459008 bytes 2023-12-10T19:05:11.769 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4800 IOs, 2234402816 bytes 2023-12-10T19:05:12.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4900 IOs, 2349638656 bytes 2023-12-10T19:05:12.274 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5000 IOs, 2390567936 bytes 2023-12-10T19:05:12.421 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5100 IOs, 2412980224 bytes 2023-12-10T19:05:12.530 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5200 IOs, 2419464192 bytes 2023-12-10T19:05:12.549 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=124 2023-12-10T19:05:12.549 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T19:05:12.549 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 124 -eq 124 ] 2023-12-10T19:05:12.549 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:05:12.549 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_15 2023-12-10T19:05:12.549 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image_and_enable_mirror cluster2 mirror image_15 snapshot 128M 2023-12-10T19:05:12.549 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:05:12.549 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:05:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:05:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:05:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_15 2023-12-10T19:05:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:05:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:05:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n snapshot ] 2023-12-10T19:05:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:05:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image cluster2 mirror image_15 128M 2023-12-10T19:05:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:05:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:05:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:05:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:05:12.551 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_15 2023-12-10T19:05:12.551 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:05:12.551 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local size=128 2023-12-10T19:05:12.551 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n 128M ] 2023-12-10T19:05:12.551 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ size=128M 2023-12-10T19:05:12.551 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:05:12.551 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_15 2023-12-10T19:05:12.596 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ is_pool_mirror_mode_image mirror 2023-12-10T19:05:12.596 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:05:12.596 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ image = image ] 2023-12-10T19:05:12.596 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:05:12.596 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ enable_mirror cluster2 mirror image_15 snapshot 2023-12-10T19:05:12.596 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:05:12.596 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:05:12.596 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_15 2023-12-10T19:05:12.596 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:05:12.596 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_15 snapshot 2023-12-10T19:05:12.733 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2023-12-10T19:05:12.733 INFO:teuthology.orchestra.run:waiting for 600 2023-12-10T19:05:12.734 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2023-12-10T19:05:12.734 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2023-12-10T19:05:12.734 DEBUG:teuthology.orchestra.run.smithi076:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2023-12-10T19:05:12.736 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2023-12-10T19:05:12.737 INFO:tasks.rbd_mirror_thrash:waiting for 90 secs before thrashing 2023-12-10T19:05:13.537 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirroring enabled 2023-12-10T19:05:13.548 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 info mirror/image_15 2023-12-10T19:05:13.595 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_15': 2023-12-10T19:05:13.595 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:05:13.595 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:05:13.596 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 1 2023-12-10T19:05:13.596 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 1317317f3d28 2023-12-10T19:05:13.596 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.1317317f3d28 2023-12-10T19:05:13.596 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:05:13.596 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering 2023-12-10T19:05:13.596 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:05:13.596 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:05:13.596 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:05:12 2023 2023-12-10T19:05:13.596 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:05:12 2023 2023-12-10T19:05:13.596 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:05:12 2023 2023-12-10T19:05:13.596 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:05:13.596 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:05:13.596 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: e5868311-cdb8-441b-a90b-45641cca9efd 2023-12-10T19:05:13.596 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: true 2023-12-10T19:05:13.599 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T19:05:13.600 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stress_write_image cluster2 mirror image_15 2023-12-10T19:05:13.600 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:05:13.600 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:05:13.600 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_15 2023-12-10T19:05:13.600 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-10T19:05:13.601 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local duration=6 2023-12-10T19:05:13.601 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T19:05:13.601 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_15 --debug-rbd=20 --debug-journaler=20 2023-12-10T19:05:13.881 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:100 IOs, 79729664 bytes 2023-12-10T19:05:14.023 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:200 IOs, 107601920 bytes 2023-12-10T19:05:14.172 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:300 IOs, 110387200 bytes 2023-12-10T19:05:14.327 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:400 IOs, 137592832 bytes 2023-12-10T19:05:14.431 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:500 IOs, 138180608 bytes 2023-12-10T19:05:14.605 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:600 IOs, 176235520 bytes 2023-12-10T19:05:14.811 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:700 IOs, 231884800 bytes 2023-12-10T19:05:14.994 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:800 IOs, 284195840 bytes 2023-12-10T19:05:15.253 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:900 IOs, 354429952 bytes 2023-12-10T19:05:15.396 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1000 IOs, 378869760 bytes 2023-12-10T19:05:15.519 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1100 IOs, 386941952 bytes 2023-12-10T19:05:15.791 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1200 IOs, 474814464 bytes 2023-12-10T19:05:15.954 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1300 IOs, 505997312 bytes 2023-12-10T19:05:16.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1400 IOs, 551745536 bytes 2023-12-10T19:05:16.284 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1500 IOs, 600495104 bytes 2023-12-10T19:05:16.429 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1600 IOs, 601085952 bytes 2023-12-10T19:05:16.672 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1700 IOs, 673579008 bytes 2023-12-10T19:05:16.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1800 IOs, 690697216 bytes 2023-12-10T19:05:16.989 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1900 IOs, 739645440 bytes 2023-12-10T19:05:17.102 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2000 IOs, 749138944 bytes 2023-12-10T19:05:17.239 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2100 IOs, 767980544 bytes 2023-12-10T19:05:17.404 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2200 IOs, 803005440 bytes 2023-12-10T19:05:17.573 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2300 IOs, 850363392 bytes 2023-12-10T19:05:17.762 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2400 IOs, 895964160 bytes 2023-12-10T19:05:18.009 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2500 IOs, 967049216 bytes 2023-12-10T19:05:18.249 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2600 IOs, 1027257344 bytes 2023-12-10T19:05:18.481 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2700 IOs, 1093014528 bytes 2023-12-10T19:05:18.642 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2800 IOs, 1130402816 bytes 2023-12-10T19:05:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2900 IOs, 1175142400 bytes 2023-12-10T19:05:18.957 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3000 IOs, 1197484032 bytes 2023-12-10T19:05:19.122 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3100 IOs, 1224026112 bytes 2023-12-10T19:05:19.245 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3200 IOs, 1230249984 bytes 2023-12-10T19:05:19.376 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3300 IOs, 1251683328 bytes 2023-12-10T19:05:19.605 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=124 2023-12-10T19:05:19.606 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T19:05:19.606 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 124 -eq 124 ] 2023-12-10T19:05:19.606 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:05:19.606 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_16 2023-12-10T19:05:19.606 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image_and_enable_mirror cluster2 mirror image_16 snapshot 128M 2023-12-10T19:05:19.606 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:05:19.606 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:05:19.606 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:05:19.606 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:05:19.606 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_16 2023-12-10T19:05:19.606 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:05:19.606 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:05:19.606 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n snapshot ] 2023-12-10T19:05:19.607 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:05:19.607 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image cluster2 mirror image_16 128M 2023-12-10T19:05:19.607 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:05:19.607 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:05:19.607 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:05:19.607 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:05:19.607 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_16 2023-12-10T19:05:19.607 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:05:19.607 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local size=128 2023-12-10T19:05:19.607 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n 128M ] 2023-12-10T19:05:19.607 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ size=128M 2023-12-10T19:05:19.607 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:05:19.607 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_16 2023-12-10T19:05:19.652 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ is_pool_mirror_mode_image mirror 2023-12-10T19:05:19.652 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:05:19.652 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ image = image ] 2023-12-10T19:05:19.652 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:05:19.652 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ enable_mirror cluster2 mirror image_16 snapshot 2023-12-10T19:05:19.653 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:05:19.653 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:05:19.653 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_16 2023-12-10T19:05:19.653 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:05:19.653 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_16 snapshot 2023-12-10T19:05:20.617 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirroring enabled 2023-12-10T19:05:20.629 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 info mirror/image_16 2023-12-10T19:05:20.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_16': 2023-12-10T19:05:20.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:05:20.676 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:05:20.676 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 1 2023-12-10T19:05:20.676 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 132b635d026c 2023-12-10T19:05:20.676 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.132b635d026c 2023-12-10T19:05:20.676 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:05:20.676 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering 2023-12-10T19:05:20.676 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:05:20.676 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:05:20.676 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:05:19 2023 2023-12-10T19:05:20.676 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:05:19 2023 2023-12-10T19:05:20.676 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:05:19 2023 2023-12-10T19:05:20.676 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:05:20.676 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:05:20.677 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: 66ef9828-72b1-4f4f-b6ef-90d776f972ce 2023-12-10T19:05:20.677 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: true 2023-12-10T19:05:20.680 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T19:05:20.680 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stress_write_image cluster2 mirror image_16 2023-12-10T19:05:20.680 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:05:20.680 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:05:20.680 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_16 2023-12-10T19:05:20.680 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-10T19:05:20.682 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local duration=9 2023-12-10T19:05:20.682 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T19:05:20.682 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_16 --debug-rbd=20 --debug-journaler=20 2023-12-10T19:05:20.905 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:100 IOs, 40799232 bytes 2023-12-10T19:05:21.038 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:200 IOs, 61701120 bytes 2023-12-10T19:05:21.221 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:300 IOs, 107841536 bytes 2023-12-10T19:05:21.361 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:400 IOs, 151867392 bytes 2023-12-10T19:05:21.564 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:500 IOs, 183998464 bytes 2023-12-10T19:05:21.670 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:600 IOs, 184627200 bytes 2023-12-10T19:05:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:700 IOs, 239080448 bytes 2023-12-10T19:05:22.070 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:800 IOs, 285618176 bytes 2023-12-10T19:05:22.427 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:900 IOs, 412982272 bytes 2023-12-10T19:05:22.606 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1000 IOs, 458022912 bytes 2023-12-10T19:05:22.790 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1100 IOs, 489419776 bytes 2023-12-10T19:05:22.910 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1200 IOs, 501016576 bytes 2023-12-10T19:05:23.074 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1300 IOs, 525876224 bytes 2023-12-10T19:05:23.285 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1400 IOs, 582184960 bytes 2023-12-10T19:05:23.473 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1500 IOs, 628837376 bytes 2023-12-10T19:05:23.610 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1600 IOs, 661362688 bytes 2023-12-10T19:05:23.759 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1700 IOs, 677632000 bytes 2023-12-10T19:05:23.893 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1800 IOs, 697550848 bytes 2023-12-10T19:05:24.175 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1900 IOs, 796323840 bytes 2023-12-10T19:05:24.411 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2000 IOs, 867069952 bytes 2023-12-10T19:05:24.587 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2100 IOs, 905585664 bytes 2023-12-10T19:05:24.737 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2200 IOs, 930338816 bytes 2023-12-10T19:05:25.007 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2300 IOs, 1010461696 bytes 2023-12-10T19:05:25.111 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2400 IOs, 1011866624 bytes 2023-12-10T19:05:25.371 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2500 IOs, 1090872320 bytes 2023-12-10T19:05:25.515 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2600 IOs, 1113295872 bytes 2023-12-10T19:05:25.706 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2700 IOs, 1165674496 bytes 2023-12-10T19:05:25.872 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2800 IOs, 1196771328 bytes 2023-12-10T19:05:25.993 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2900 IOs, 1208252416 bytes 2023-12-10T19:05:26.213 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3000 IOs, 1269320704 bytes 2023-12-10T19:05:26.413 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3100 IOs, 1324632064 bytes 2023-12-10T19:05:26.544 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3200 IOs, 1344669696 bytes 2023-12-10T19:05:26.646 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3300 IOs, 1345289216 bytes 2023-12-10T19:05:26.817 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3400 IOs, 1378359296 bytes 2023-12-10T19:05:26.993 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3500 IOs, 1408827392 bytes 2023-12-10T19:05:27.175 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3600 IOs, 1457498112 bytes 2023-12-10T19:05:27.495 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3700 IOs, 1562812416 bytes 2023-12-10T19:05:27.603 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3800 IOs, 1568494592 bytes 2023-12-10T19:05:27.758 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3900 IOs, 1604210688 bytes 2023-12-10T19:05:27.928 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4000 IOs, 1635061760 bytes 2023-12-10T19:05:28.072 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4100 IOs, 1646809088 bytes 2023-12-10T19:05:28.267 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4200 IOs, 1692759040 bytes 2023-12-10T19:05:28.419 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4300 IOs, 1723655168 bytes 2023-12-10T19:05:28.522 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4400 IOs, 1727934464 bytes 2023-12-10T19:05:28.632 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4500 IOs, 1734746112 bytes 2023-12-10T19:05:28.808 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4600 IOs, 1772517376 bytes 2023-12-10T19:05:28.993 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4700 IOs, 1809723392 bytes 2023-12-10T19:05:29.246 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4800 IOs, 1890989056 bytes 2023-12-10T19:05:29.468 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4900 IOs, 1947227136 bytes 2023-12-10T19:05:29.648 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5000 IOs, 1984372736 bytes 2023-12-10T19:05:29.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=124 2023-12-10T19:05:29.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T19:05:29.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 124 -eq 124 ] 2023-12-10T19:05:29.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:05:29.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_17 2023-12-10T19:05:29.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image_and_enable_mirror cluster2 mirror image_17 snapshot 128M 2023-12-10T19:05:29.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:05:29.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:05:29.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:05:29.687 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:05:29.687 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_17 2023-12-10T19:05:29.687 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:05:29.687 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:05:29.687 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n snapshot ] 2023-12-10T19:05:29.687 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:05:29.687 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image cluster2 mirror image_17 128M 2023-12-10T19:05:29.687 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:05:29.687 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:05:29.687 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:05:29.687 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:05:29.687 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_17 2023-12-10T19:05:29.687 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:05:29.687 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local size=128 2023-12-10T19:05:29.687 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n 128M ] 2023-12-10T19:05:29.687 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ size=128M 2023-12-10T19:05:29.688 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:05:29.688 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_17 2023-12-10T19:05:29.733 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ is_pool_mirror_mode_image mirror 2023-12-10T19:05:29.734 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:05:29.734 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ image = image ] 2023-12-10T19:05:29.734 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:05:29.734 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ enable_mirror cluster2 mirror image_17 snapshot 2023-12-10T19:05:29.734 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:05:29.734 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:05:29.734 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_17 2023-12-10T19:05:29.734 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:05:29.734 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_17 snapshot 2023-12-10T19:05:30.516 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirroring enabled 2023-12-10T19:05:30.526 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 info mirror/image_17 2023-12-10T19:05:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_17': 2023-12-10T19:05:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:05:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:05:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 1 2023-12-10T19:05:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 13373d7fecb6 2023-12-10T19:05:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.13373d7fecb6 2023-12-10T19:05:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:05:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering 2023-12-10T19:05:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:05:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:05:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:05:29 2023 2023-12-10T19:05:30.572 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:05:29 2023 2023-12-10T19:05:30.572 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:05:29 2023 2023-12-10T19:05:30.572 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:05:30.572 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:05:30.572 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: a44b2376-7333-4134-9c68-c71acef2b46e 2023-12-10T19:05:30.572 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: true 2023-12-10T19:05:30.575 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T19:05:30.575 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stress_write_image cluster2 mirror image_17 2023-12-10T19:05:30.575 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:05:30.575 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:05:30.575 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_17 2023-12-10T19:05:30.575 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-10T19:05:30.576 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local duration=8 2023-12-10T19:05:30.576 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T19:05:30.576 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_17 --debug-rbd=20 --debug-journaler=20 2023-12-10T19:05:30.810 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:100 IOs, 52597760 bytes 2023-12-10T19:05:30.909 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:200 IOs, 53240832 bytes 2023-12-10T19:05:31.117 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:300 IOs, 125860864 bytes 2023-12-10T19:05:31.310 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:400 IOs, 181412864 bytes 2023-12-10T19:05:31.508 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:500 IOs, 232125440 bytes 2023-12-10T19:05:31.682 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:600 IOs, 252248064 bytes 2023-12-10T19:05:31.818 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:700 IOs, 268979200 bytes 2023-12-10T19:05:31.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:800 IOs, 308530176 bytes 2023-12-10T19:05:32.193 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:900 IOs, 355421184 bytes 2023-12-10T19:05:32.470 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1000 IOs, 447370240 bytes 2023-12-10T19:05:32.632 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1100 IOs, 492577792 bytes 2023-12-10T19:05:32.850 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1200 IOs, 540420096 bytes 2023-12-10T19:05:33.065 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1300 IOs, 601037824 bytes 2023-12-10T19:05:33.197 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1400 IOs, 622046208 bytes 2023-12-10T19:05:33.404 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1500 IOs, 695932928 bytes 2023-12-10T19:05:33.606 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1600 IOs, 731528192 bytes 2023-12-10T19:05:33.755 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1700 IOs, 752492544 bytes 2023-12-10T19:05:33.881 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1800 IOs, 766242816 bytes 2023-12-10T19:05:34.086 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1900 IOs, 841249792 bytes 2023-12-10T19:05:34.257 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2000 IOs, 881362944 bytes 2023-12-10T19:05:34.461 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2100 IOs, 947091456 bytes 2023-12-10T19:05:34.652 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2200 IOs, 970735616 bytes 2023-12-10T19:05:34.849 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2300 IOs, 1006935040 bytes 2023-12-10T19:05:35.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2400 IOs, 1068894208 bytes 2023-12-10T19:05:35.192 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2500 IOs, 1087012864 bytes 2023-12-10T19:05:35.313 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2600 IOs, 1087620096 bytes 2023-12-10T19:05:35.539 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2700 IOs, 1148636160 bytes 2023-12-10T19:05:35.778 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2800 IOs, 1212913664 bytes 2023-12-10T19:05:35.948 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2900 IOs, 1237454848 bytes 2023-12-10T19:05:36.184 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3000 IOs, 1302715392 bytes 2023-12-10T19:05:36.324 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3100 IOs, 1325330432 bytes 2023-12-10T19:05:36.514 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3200 IOs, 1387088896 bytes 2023-12-10T19:05:36.661 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3300 IOs, 1387696128 bytes 2023-12-10T19:05:36.892 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3400 IOs, 1456448512 bytes 2023-12-10T19:05:37.051 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3500 IOs, 1477488640 bytes 2023-12-10T19:05:37.330 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3600 IOs, 1563828224 bytes 2023-12-10T19:05:37.562 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3700 IOs, 1625013248 bytes 2023-12-10T19:05:37.714 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3800 IOs, 1646586880 bytes 2023-12-10T19:05:37.827 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3900 IOs, 1651589120 bytes 2023-12-10T19:05:37.989 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4000 IOs, 1676270592 bytes 2023-12-10T19:05:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4100 IOs, 1744720896 bytes 2023-12-10T19:05:38.404 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4200 IOs, 1779527680 bytes 2023-12-10T19:05:38.538 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4300 IOs, 1784155136 bytes 2023-12-10T19:05:38.580 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=124 2023-12-10T19:05:38.580 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T19:05:38.580 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 124 -eq 124 ] 2023-12-10T19:05:38.580 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:05:38.580 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_18 2023-12-10T19:05:38.580 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image_and_enable_mirror cluster2 mirror image_18 snapshot 128M 2023-12-10T19:05:38.581 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:05:38.581 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:05:38.581 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:05:38.581 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:05:38.581 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_18 2023-12-10T19:05:38.581 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:05:38.581 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:05:38.581 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n snapshot ] 2023-12-10T19:05:38.581 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:05:38.581 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image cluster2 mirror image_18 128M 2023-12-10T19:05:38.581 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:05:38.581 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:05:38.581 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:05:38.581 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:05:38.582 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_18 2023-12-10T19:05:38.582 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:05:38.582 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local size=128 2023-12-10T19:05:38.582 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n 128M ] 2023-12-10T19:05:38.582 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ size=128M 2023-12-10T19:05:38.582 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:05:38.582 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_18 2023-12-10T19:05:38.626 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ is_pool_mirror_mode_image mirror 2023-12-10T19:05:38.627 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:05:38.627 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ image = image ] 2023-12-10T19:05:38.627 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:05:38.627 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ enable_mirror cluster2 mirror image_18 snapshot 2023-12-10T19:05:38.627 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:05:38.627 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:05:38.627 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_18 2023-12-10T19:05:38.627 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:05:38.627 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_18 snapshot 2023-12-10T19:05:39.498 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirroring enabled 2023-12-10T19:05:39.511 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 info mirror/image_18 2023-12-10T19:05:39.555 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_18': 2023-12-10T19:05:39.556 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:05:39.556 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:05:39.556 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 1 2023-12-10T19:05:39.556 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 13432df749a7 2023-12-10T19:05:39.556 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.13432df749a7 2023-12-10T19:05:39.556 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:05:39.556 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering 2023-12-10T19:05:39.556 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:05:39.556 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:05:39.556 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:05:38 2023 2023-12-10T19:05:39.556 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:05:38 2023 2023-12-10T19:05:39.556 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:05:38 2023 2023-12-10T19:05:39.556 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:05:39.556 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:05:39.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: 013b5e35-24cb-486f-93bf-0f941ebb1c1d 2023-12-10T19:05:39.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: true 2023-12-10T19:05:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T19:05:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stress_write_image cluster2 mirror image_18 2023-12-10T19:05:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:05:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:05:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_18 2023-12-10T19:05:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-10T19:05:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local duration=7 2023-12-10T19:05:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T19:05:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_18 --debug-rbd=20 --debug-journaler=20 2023-12-10T19:05:39.798 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:100 IOs, 44401664 bytes 2023-12-10T19:05:40.023 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:200 IOs, 99580928 bytes 2023-12-10T19:05:40.221 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:300 IOs, 148291584 bytes 2023-12-10T19:05:40.379 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:400 IOs, 181750784 bytes 2023-12-10T19:05:40.553 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:500 IOs, 208351232 bytes 2023-12-10T19:05:40.740 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:600 IOs, 248166400 bytes 2023-12-10T19:05:40.849 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:700 IOs, 250867712 bytes 2023-12-10T19:05:41.045 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:800 IOs, 312197120 bytes 2023-12-10T19:05:41.190 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:900 IOs, 329226240 bytes 2023-12-10T19:05:41.360 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1000 IOs, 356261888 bytes 2023-12-10T19:05:41.469 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1100 IOs, 362461184 bytes 2023-12-10T19:05:41.571 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1200 IOs, 363212800 bytes 2023-12-10T19:05:41.725 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1300 IOs, 384723968 bytes 2023-12-10T19:05:42.055 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1400 IOs, 494398464 bytes 2023-12-10T19:05:42.211 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1500 IOs, 523983872 bytes 2023-12-10T19:05:42.450 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1600 IOs, 591262720 bytes 2023-12-10T19:05:42.705 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1700 IOs, 671694848 bytes 2023-12-10T19:05:43.000 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1800 IOs, 758638592 bytes 2023-12-10T19:05:43.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1900 IOs, 781452288 bytes 2023-12-10T19:05:43.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2000 IOs, 830554112 bytes 2023-12-10T19:05:43.563 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2100 IOs, 886349824 bytes 2023-12-10T19:05:43.688 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2200 IOs, 907449344 bytes 2023-12-10T19:05:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2300 IOs, 975413248 bytes 2023-12-10T19:05:44.159 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2400 IOs, 1029060608 bytes 2023-12-10T19:05:44.349 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2500 IOs, 1081036800 bytes 2023-12-10T19:05:44.508 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2600 IOs, 1129108480 bytes 2023-12-10T19:05:44.768 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2700 IOs, 1203615744 bytes 2023-12-10T19:05:44.946 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2800 IOs, 1234138112 bytes 2023-12-10T19:05:45.190 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2900 IOs, 1298913280 bytes 2023-12-10T19:05:45.406 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3000 IOs, 1360563200 bytes 2023-12-10T19:05:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3100 IOs, 1407105024 bytes 2023-12-10T19:05:45.741 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3200 IOs, 1423520768 bytes 2023-12-10T19:05:45.843 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3300 IOs, 1424145408 bytes 2023-12-10T19:05:45.970 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3400 IOs, 1439411200 bytes 2023-12-10T19:05:46.179 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3500 IOs, 1490351104 bytes 2023-12-10T19:05:46.375 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3600 IOs, 1541080064 bytes 2023-12-10T19:05:46.566 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=124 2023-12-10T19:05:46.566 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T19:05:46.566 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 124 -eq 124 ] 2023-12-10T19:05:46.566 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:05:46.566 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_19 2023-12-10T19:05:46.566 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image_and_enable_mirror cluster2 mirror image_19 snapshot 128M 2023-12-10T19:05:46.566 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:05:46.566 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:05:46.567 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:05:46.567 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:05:46.567 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_19 2023-12-10T19:05:46.567 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:05:46.567 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:05:46.567 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n snapshot ] 2023-12-10T19:05:46.567 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:05:46.567 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image cluster2 mirror image_19 128M 2023-12-10T19:05:46.567 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:05:46.567 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:05:46.567 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:05:46.567 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:05:46.567 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_19 2023-12-10T19:05:46.567 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:05:46.567 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local size=128 2023-12-10T19:05:46.568 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n 128M ] 2023-12-10T19:05:46.568 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ size=128M 2023-12-10T19:05:46.568 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:05:46.568 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_19 2023-12-10T19:05:46.612 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ is_pool_mirror_mode_image mirror 2023-12-10T19:05:46.612 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:05:46.612 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ image = image ] 2023-12-10T19:05:46.612 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:05:46.612 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ enable_mirror cluster2 mirror image_19 snapshot 2023-12-10T19:05:46.612 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:05:46.612 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:05:46.612 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_19 2023-12-10T19:05:46.612 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:05:46.612 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_19 snapshot 2023-12-10T19:05:47.484 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirroring enabled 2023-12-10T19:05:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 info mirror/image_19 2023-12-10T19:05:47.540 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_19': 2023-12-10T19:05:47.540 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:05:47.540 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:05:47.540 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 1 2023-12-10T19:05:47.540 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 134f31693d64 2023-12-10T19:05:47.540 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.134f31693d64 2023-12-10T19:05:47.540 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:05:47.540 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering 2023-12-10T19:05:47.540 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:05:47.540 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:05:47.540 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:05:46 2023 2023-12-10T19:05:47.540 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:05:46 2023 2023-12-10T19:05:47.540 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:05:46 2023 2023-12-10T19:05:47.540 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:05:47.541 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:05:47.541 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: 9885ccb8-b168-4992-a2b7-1fe906e83484 2023-12-10T19:05:47.541 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: true 2023-12-10T19:05:47.544 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T19:05:47.544 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stress_write_image cluster2 mirror image_19 2023-12-10T19:05:47.544 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:05:47.544 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:05:47.544 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_19 2023-12-10T19:05:47.544 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-10T19:05:47.545 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local duration=14 2023-12-10T19:05:47.545 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T19:05:47.545 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_19 --debug-rbd=20 --debug-journaler=20 2023-12-10T19:05:47.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:100 IOs, 18415616 bytes 2023-12-10T19:05:47.847 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:200 IOs, 36604928 bytes 2023-12-10T19:05:48.051 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:300 IOs, 81952768 bytes 2023-12-10T19:05:48.345 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:400 IOs, 173276160 bytes 2023-12-10T19:05:48.462 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:500 IOs, 187365376 bytes 2023-12-10T19:05:48.598 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:600 IOs, 212586496 bytes 2023-12-10T19:05:48.694 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:700 IOs, 213214208 bytes 2023-12-10T19:05:48.851 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:800 IOs, 243980288 bytes 2023-12-10T19:05:49.030 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:900 IOs, 282976256 bytes 2023-12-10T19:05:49.225 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1000 IOs, 351471616 bytes 2023-12-10T19:05:49.478 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1100 IOs, 416556032 bytes 2023-12-10T19:05:49.588 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1200 IOs, 423696384 bytes 2023-12-10T19:05:49.811 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1300 IOs, 492585984 bytes 2023-12-10T19:05:49.998 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1400 IOs, 535248896 bytes 2023-12-10T19:05:50.143 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1500 IOs, 554752000 bytes 2023-12-10T19:05:50.362 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1600 IOs, 608325632 bytes 2023-12-10T19:05:50.528 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1700 IOs, 641588224 bytes 2023-12-10T19:05:50.662 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1800 IOs, 663724032 bytes 2023-12-10T19:05:50.903 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1900 IOs, 732705792 bytes 2023-12-10T19:05:51.064 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2000 IOs, 762219520 bytes 2023-12-10T19:05:51.300 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2100 IOs, 831734784 bytes 2023-12-10T19:05:51.467 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2200 IOs, 865372160 bytes 2023-12-10T19:05:51.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2300 IOs, 924265472 bytes 2023-12-10T19:05:51.778 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2400 IOs, 924856320 bytes 2023-12-10T19:05:51.960 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2500 IOs, 966753280 bytes 2023-12-10T19:05:52.179 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2600 IOs, 1021025280 bytes 2023-12-10T19:05:52.304 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2700 IOs, 1032357888 bytes 2023-12-10T19:05:52.437 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2800 IOs, 1051063296 bytes 2023-12-10T19:05:52.618 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2900 IOs, 1084530688 bytes 2023-12-10T19:05:52.800 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3000 IOs, 1122610176 bytes 2023-12-10T19:05:52.924 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3100 IOs, 1148736512 bytes 2023-12-10T19:05:53.166 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3200 IOs, 1233999872 bytes 2023-12-10T19:05:53.348 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3300 IOs, 1253416960 bytes 2023-12-10T19:05:53.509 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3400 IOs, 1277860864 bytes 2023-12-10T19:05:53.734 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3500 IOs, 1340104704 bytes 2023-12-10T19:05:53.896 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3600 IOs, 1370428416 bytes 2023-12-10T19:05:54.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3700 IOs, 1400832000 bytes 2023-12-10T19:05:54.321 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3800 IOs, 1474029568 bytes 2023-12-10T19:05:54.578 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3900 IOs, 1549029376 bytes 2023-12-10T19:05:54.820 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4000 IOs, 1621024768 bytes 2023-12-10T19:05:55.017 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4100 IOs, 1668013056 bytes 2023-12-10T19:05:55.137 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4200 IOs, 1676187648 bytes 2023-12-10T19:05:55.323 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4300 IOs, 1708315648 bytes 2023-12-10T19:05:55.484 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4400 IOs, 1741023232 bytes 2023-12-10T19:05:55.637 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4500 IOs, 1768862720 bytes 2023-12-10T19:05:55.826 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4600 IOs, 1810461696 bytes 2023-12-10T19:05:56.125 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4700 IOs, 1904769024 bytes 2023-12-10T19:05:56.258 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4800 IOs, 1939472384 bytes 2023-12-10T19:05:56.569 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4900 IOs, 2014964736 bytes 2023-12-10T19:05:56.733 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5000 IOs, 2043490304 bytes 2023-12-10T19:05:56.879 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5100 IOs, 2065221632 bytes 2023-12-10T19:05:57.043 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5200 IOs, 2093839360 bytes 2023-12-10T19:05:57.249 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5300 IOs, 2165403648 bytes 2023-12-10T19:05:57.382 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5400 IOs, 2165992448 bytes 2023-12-10T19:05:57.569 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5500 IOs, 2206104576 bytes 2023-12-10T19:05:57.817 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5600 IOs, 2269783040 bytes 2023-12-10T19:05:57.973 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5700 IOs, 2299403264 bytes 2023-12-10T19:05:58.186 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5800 IOs, 2369605632 bytes 2023-12-10T19:05:58.411 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5900 IOs, 2414439424 bytes 2023-12-10T19:05:58.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6000 IOs, 2495855616 bytes 2023-12-10T19:05:58.921 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6100 IOs, 2560120832 bytes 2023-12-10T19:05:59.160 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6200 IOs, 2633176064 bytes 2023-12-10T19:05:59.338 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6300 IOs, 2678176768 bytes 2023-12-10T19:05:59.565 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6400 IOs, 2735277056 bytes 2023-12-10T19:05:59.859 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6500 IOs, 2830514176 bytes 2023-12-10T19:06:00.066 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6600 IOs, 2907668480 bytes 2023-12-10T19:06:00.274 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6700 IOs, 2958578688 bytes 2023-12-10T19:06:00.427 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6800 IOs, 2969247744 bytes 2023-12-10T19:06:00.683 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6900 IOs, 3042962432 bytes 2023-12-10T19:06:00.824 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:7000 IOs, 3060638720 bytes 2023-12-10T19:06:01.096 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:7100 IOs, 3148845056 bytes 2023-12-10T19:06:01.327 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:7200 IOs, 3224370176 bytes 2023-12-10T19:06:01.527 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:7300 IOs, 3259187200 bytes 2023-12-10T19:06:01.549 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=124 2023-12-10T19:06:01.550 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T19:06:01.550 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 124 -eq 124 ] 2023-12-10T19:06:01.550 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:06:01.550 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_20 2023-12-10T19:06:01.550 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image_and_enable_mirror cluster2 mirror image_20 snapshot 128M 2023-12-10T19:06:01.550 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:06:01.550 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:06:01.550 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:06:01.550 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:06:01.550 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_20 2023-12-10T19:06:01.550 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:06:01.550 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:06:01.550 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n snapshot ] 2023-12-10T19:06:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:06:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image cluster2 mirror image_20 128M 2023-12-10T19:06:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:06:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:06:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:06:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:06:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_20 2023-12-10T19:06:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:06:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local size=128 2023-12-10T19:06:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n 128M ] 2023-12-10T19:06:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ size=128M 2023-12-10T19:06:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:06:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_20 2023-12-10T19:06:01.597 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ is_pool_mirror_mode_image mirror 2023-12-10T19:06:01.597 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:06:01.597 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ image = image ] 2023-12-10T19:06:01.597 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:06:01.597 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ enable_mirror cluster2 mirror image_20 snapshot 2023-12-10T19:06:01.598 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:06:01.598 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:06:01.598 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_20 2023-12-10T19:06:01.598 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:06:01.598 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_20 snapshot 2023-12-10T19:06:02.503 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirroring enabled 2023-12-10T19:06:02.517 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 info mirror/image_20 2023-12-10T19:06:02.564 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_20': 2023-12-10T19:06:02.564 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:06:02.564 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:06:02.564 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 1 2023-12-10T19:06:02.564 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 135b9423b4b6 2023-12-10T19:06:02.565 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.135b9423b4b6 2023-12-10T19:06:02.565 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:06:02.565 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering 2023-12-10T19:06:02.565 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:06:02.565 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:06:02.565 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:06:01 2023 2023-12-10T19:06:02.565 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:06:01 2023 2023-12-10T19:06:02.565 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:06:01 2023 2023-12-10T19:06:02.566 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:06:02.566 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:06:02.566 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: ad3d186e-bfbd-40cc-b37d-73be0b0f0a7d 2023-12-10T19:06:02.566 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: true 2023-12-10T19:06:02.568 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T19:06:02.568 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stress_write_image cluster2 mirror image_20 2023-12-10T19:06:02.568 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:06:02.569 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:06:02.569 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_20 2023-12-10T19:06:02.569 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-10T19:06:02.569 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local duration=10 2023-12-10T19:06:02.569 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T19:06:02.570 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_20 --debug-rbd=20 --debug-journaler=20 2023-12-10T19:06:02.862 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:100 IOs, 73534464 bytes 2023-12-10T19:06:02.966 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:200 IOs, 75225088 bytes 2023-12-10T19:06:03.134 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:300 IOs, 116861952 bytes 2023-12-10T19:06:03.261 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:400 IOs, 122114048 bytes 2023-12-10T19:06:03.412 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:500 IOs, 147798016 bytes 2023-12-10T19:06:03.605 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:600 IOs, 202340352 bytes 2023-12-10T19:06:03.748 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:700 IOs, 220001280 bytes 2023-12-10T19:06:03.958 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:800 IOs, 287469568 bytes 2023-12-10T19:06:04.125 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:900 IOs, 305977344 bytes 2023-12-10T19:06:04.269 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1000 IOs, 327989248 bytes 2023-12-10T19:06:04.398 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1100 IOs, 348681216 bytes 2023-12-10T19:06:04.547 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1200 IOs, 366835712 bytes 2023-12-10T19:06:04.712 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1300 IOs, 396864512 bytes 2023-12-10T19:06:04.869 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1400 IOs, 420260864 bytes 2023-12-10T19:06:05.111 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1500 IOs, 498109440 bytes 2023-12-10T19:06:05.420 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1600 IOs, 598805504 bytes 2023-12-10T19:06:05.593 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1700 IOs, 634140672 bytes 2023-12-10T19:06:05.889 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1800 IOs, 736544768 bytes 2023-12-10T19:06:06.234 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1900 IOs, 869038080 bytes 2023-12-10T19:06:06.475 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2000 IOs, 921329664 bytes 2023-12-10T19:06:06.643 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2100 IOs, 958092288 bytes 2023-12-10T19:06:06.893 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2200 IOs, 1050380288 bytes 2023-12-10T19:06:07.278 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2300 IOs, 1174037504 bytes 2023-12-10T19:06:07.445 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2400 IOs, 1218388992 bytes 2023-12-10T19:06:07.576 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2500 IOs, 1224224768 bytes 2023-12-10T19:06:07.749 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2600 IOs, 1265619968 bytes 2023-12-10T19:06:07.958 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2700 IOs, 1318685696 bytes 2023-12-10T19:06:08.129 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2800 IOs, 1344548864 bytes 2023-12-10T19:06:08.310 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2900 IOs, 1391008768 bytes 2023-12-10T19:06:08.446 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3000 IOs, 1408124928 bytes 2023-12-10T19:06:08.619 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3100 IOs, 1449539584 bytes 2023-12-10T19:06:08.925 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3200 IOs, 1549547520 bytes 2023-12-10T19:06:09.170 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3300 IOs, 1617606656 bytes 2023-12-10T19:06:09.413 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3400 IOs, 1684715520 bytes 2023-12-10T19:06:09.567 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3500 IOs, 1701742592 bytes 2023-12-10T19:06:09.722 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3600 IOs, 1743910912 bytes 2023-12-10T19:06:09.906 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3700 IOs, 1763993600 bytes 2023-12-10T19:06:10.074 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3800 IOs, 1797190656 bytes 2023-12-10T19:06:10.301 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3900 IOs, 1898402816 bytes 2023-12-10T19:06:10.487 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4000 IOs, 1898997760 bytes 2023-12-10T19:06:10.676 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4100 IOs, 1971566592 bytes 2023-12-10T19:06:10.964 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4200 IOs, 2036913152 bytes 2023-12-10T19:06:11.198 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4300 IOs, 2094844928 bytes 2023-12-10T19:06:11.378 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4400 IOs, 2128456704 bytes 2023-12-10T19:06:11.490 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4500 IOs, 2131348480 bytes 2023-12-10T19:06:11.605 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4600 IOs, 2131994624 bytes 2023-12-10T19:06:11.717 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4700 IOs, 2138881024 bytes 2023-12-10T19:06:11.914 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4800 IOs, 2188456960 bytes 2023-12-10T19:06:12.042 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4900 IOs, 2221103104 bytes 2023-12-10T19:06:12.236 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5000 IOs, 2248738816 bytes 2023-12-10T19:06:12.331 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5100 IOs, 2249347072 bytes 2023-12-10T19:06:12.500 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5200 IOs, 2285955072 bytes 2023-12-10T19:06:12.573 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=124 2023-12-10T19:06:12.573 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T19:06:12.573 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 124 -eq 124 ] 2023-12-10T19:06:12.574 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:06:12.574 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_21 2023-12-10T19:06:12.574 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image_and_enable_mirror cluster2 mirror image_21 snapshot 128M 2023-12-10T19:06:12.574 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:06:12.574 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:06:12.574 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:06:12.574 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:06:12.574 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_21 2023-12-10T19:06:12.574 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:06:12.574 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:06:12.574 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n snapshot ] 2023-12-10T19:06:12.574 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:06:12.574 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image cluster2 mirror image_21 128M 2023-12-10T19:06:12.575 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:06:12.575 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:06:12.575 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:06:12.575 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:06:12.575 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_21 2023-12-10T19:06:12.575 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:06:12.575 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local size=128 2023-12-10T19:06:12.575 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n 128M ] 2023-12-10T19:06:12.575 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ size=128M 2023-12-10T19:06:12.575 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:06:12.575 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_21 2023-12-10T19:06:12.622 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ is_pool_mirror_mode_image mirror 2023-12-10T19:06:12.623 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:06:12.623 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ image = image ] 2023-12-10T19:06:12.623 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:06:12.623 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ enable_mirror cluster2 mirror image_21 snapshot 2023-12-10T19:06:12.623 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:06:12.623 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:06:12.623 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_21 2023-12-10T19:06:12.623 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:06:12.623 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_21 snapshot 2023-12-10T19:06:13.544 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirroring enabled 2023-12-10T19:06:13.558 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 info mirror/image_21 2023-12-10T19:06:13.605 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_21': 2023-12-10T19:06:13.605 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:06:13.605 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:06:13.606 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 1 2023-12-10T19:06:13.606 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 1367b32740f1 2023-12-10T19:06:13.606 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.1367b32740f1 2023-12-10T19:06:13.606 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:06:13.606 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering 2023-12-10T19:06:13.606 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:06:13.606 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:06:13.606 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:06:12 2023 2023-12-10T19:06:13.606 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:06:12 2023 2023-12-10T19:06:13.606 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:06:12 2023 2023-12-10T19:06:13.606 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:06:13.606 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:06:13.606 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: 341379c3-72c1-42e0-8bda-bc8752bf5c19 2023-12-10T19:06:13.606 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: true 2023-12-10T19:06:13.609 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T19:06:13.609 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stress_write_image cluster2 mirror image_21 2023-12-10T19:06:13.609 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:06:13.609 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:06:13.609 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_21 2023-12-10T19:06:13.609 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-10T19:06:13.610 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local duration=10 2023-12-10T19:06:13.611 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T19:06:13.611 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_21 --debug-rbd=20 --debug-journaler=20 2023-12-10T19:06:13.803 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:100 IOs, 23024640 bytes 2023-12-10T19:06:13.947 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:200 IOs, 44935168 bytes 2023-12-10T19:06:14.109 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:300 IOs, 90929152 bytes 2023-12-10T19:06:14.376 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:400 IOs, 167457792 bytes 2023-12-10T19:06:14.542 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:500 IOs, 178138112 bytes 2023-12-10T19:06:14.766 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:600 IOs, 237048832 bytes 2023-12-10T19:06:15.005 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:700 IOs, 312697856 bytes 2023-12-10T19:06:15.197 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:800 IOs, 354530304 bytes 2023-12-10T19:06:15.426 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:900 IOs, 413209600 bytes 2023-12-10T19:06:15.595 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1000 IOs, 448721920 bytes 2023-12-10T19:06:15.719 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1100 IOs, 494280704 bytes 2023-12-10T19:06:16.060 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1200 IOs, 572384256 bytes 2023-12-10T19:06:16.271 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1300 IOs, 623607808 bytes 2023-12-10T19:06:16.414 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1400 IOs, 635761664 bytes 2023-12-10T19:06:16.653 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1500 IOs, 702943232 bytes 2023-12-10T19:06:16.779 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1600 IOs, 716566528 bytes 2023-12-10T19:06:16.956 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1700 IOs, 747291648 bytes 2023-12-10T19:06:17.102 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1800 IOs, 770841600 bytes 2023-12-10T19:06:17.288 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1900 IOs, 810786816 bytes 2023-12-10T19:06:17.484 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2000 IOs, 857952256 bytes 2023-12-10T19:06:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2100 IOs, 889160704 bytes 2023-12-10T19:06:17.813 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2200 IOs, 920822784 bytes 2023-12-10T19:06:17.981 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2300 IOs, 953854976 bytes 2023-12-10T19:06:18.218 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2400 IOs, 1040234496 bytes 2023-12-10T19:06:18.504 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2500 IOs, 1108051968 bytes 2023-12-10T19:06:18.756 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2600 IOs, 1185029120 bytes 2023-12-10T19:06:18.907 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2700 IOs, 1211377664 bytes 2023-12-10T19:06:19.064 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2800 IOs, 1244691456 bytes 2023-12-10T19:06:19.302 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2900 IOs, 1315445760 bytes 2023-12-10T19:06:19.421 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3000 IOs, 1322158080 bytes 2023-12-10T19:06:19.643 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3100 IOs, 1388834816 bytes 2023-12-10T19:06:19.799 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3200 IOs, 1415650304 bytes 2023-12-10T19:06:20.005 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3300 IOs, 1466668032 bytes 2023-12-10T19:06:20.108 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3400 IOs, 1467275264 bytes 2023-12-10T19:06:20.239 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3500 IOs, 1482303488 bytes 2023-12-10T19:06:20.344 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3600 IOs, 1482909696 bytes 2023-12-10T19:06:20.514 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3700 IOs, 1534646272 bytes 2023-12-10T19:06:20.658 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3800 IOs, 1538230272 bytes 2023-12-10T19:06:20.892 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3900 IOs, 1604068352 bytes 2023-12-10T19:06:21.088 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4000 IOs, 1645251584 bytes 2023-12-10T19:06:21.336 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4100 IOs, 1716600832 bytes 2023-12-10T19:06:21.583 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4200 IOs, 1784923136 bytes 2023-12-10T19:06:21.946 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4300 IOs, 1913959424 bytes 2023-12-10T19:06:22.171 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4400 IOs, 1970836480 bytes 2023-12-10T19:06:22.312 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4500 IOs, 1997259776 bytes 2023-12-10T19:06:22.479 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4600 IOs, 2050032640 bytes 2023-12-10T19:06:22.731 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4700 IOs, 2109059072 bytes 2023-12-10T19:06:22.956 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4800 IOs, 2174985216 bytes 2023-12-10T19:06:23.188 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4900 IOs, 2248522752 bytes 2023-12-10T19:06:23.485 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5000 IOs, 2334654464 bytes 2023-12-10T19:06:23.615 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=124 2023-12-10T19:06:23.615 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T19:06:23.615 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 124 -eq 124 ] 2023-12-10T19:06:23.615 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:06:23.615 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_22 2023-12-10T19:06:23.615 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image_and_enable_mirror cluster2 mirror image_22 snapshot 128M 2023-12-10T19:06:23.615 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:06:23.615 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:06:23.616 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:06:23.616 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:06:23.616 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_22 2023-12-10T19:06:23.616 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:06:23.616 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:06:23.616 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n snapshot ] 2023-12-10T19:06:23.616 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:06:23.616 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image cluster2 mirror image_22 128M 2023-12-10T19:06:23.616 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:06:23.616 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:06:23.616 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:06:23.616 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:06:23.616 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_22 2023-12-10T19:06:23.616 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:06:23.617 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local size=128 2023-12-10T19:06:23.617 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n 128M ] 2023-12-10T19:06:23.617 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ size=128M 2023-12-10T19:06:23.617 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:06:23.617 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_22 2023-12-10T19:06:23.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ is_pool_mirror_mode_image mirror 2023-12-10T19:06:23.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:06:23.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ image = image ] 2023-12-10T19:06:23.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:06:23.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ enable_mirror cluster2 mirror image_22 snapshot 2023-12-10T19:06:23.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:06:23.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:06:23.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_22 2023-12-10T19:06:23.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:06:23.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_22 snapshot 2023-12-10T19:06:24.624 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirroring enabled 2023-12-10T19:06:24.635 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 info mirror/image_22 2023-12-10T19:06:24.888 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_22': 2023-12-10T19:06:24.889 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:06:24.889 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:06:24.889 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 1 2023-12-10T19:06:24.889 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 1373e09e84d4 2023-12-10T19:06:24.889 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.1373e09e84d4 2023-12-10T19:06:24.889 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:06:24.889 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering 2023-12-10T19:06:24.889 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:06:24.889 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:06:24.889 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:06:23 2023 2023-12-10T19:06:24.889 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:06:23 2023 2023-12-10T19:06:24.889 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:06:23 2023 2023-12-10T19:06:24.889 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:06:24.890 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:06:24.890 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: 5f3f027e-9f9a-463e-be32-1e7c54f42e4e 2023-12-10T19:06:24.890 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: true 2023-12-10T19:06:24.895 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T19:06:24.895 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stress_write_image cluster2 mirror image_22 2023-12-10T19:06:24.896 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:06:24.896 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:06:24.896 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_22 2023-12-10T19:06:24.896 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-10T19:06:24.898 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local duration=11 2023-12-10T19:06:24.898 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T19:06:24.898 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_22 --debug-rbd=20 --debug-journaler=20 2023-12-10T19:06:25.088 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:100 IOs, 10699776 bytes 2023-12-10T19:06:25.259 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:200 IOs, 50263040 bytes 2023-12-10T19:06:25.441 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:300 IOs, 87150592 bytes 2023-12-10T19:06:25.553 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:400 IOs, 94426112 bytes 2023-12-10T19:06:25.738 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-12-10T19:06:25.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-12-10T19:06:25.738 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2023-12-10T19:06:25.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr:2023-12-10T19:06:25.744+0000 7f395d664640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 45741) UID: 1000 2023-12-10T19:06:25.791 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:500 IOs, 166306816 bytes 2023-12-10T19:06:25.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:600 IOs, 203469824 bytes 2023-12-10T19:06:26.216 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:700 IOs, 269189120 bytes 2023-12-10T19:06:26.383 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:800 IOs, 301402112 bytes 2023-12-10T19:06:26.589 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:900 IOs, 353053696 bytes 2023-12-10T19:06:26.820 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1000 IOs, 412342272 bytes 2023-12-10T19:06:26.998 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1100 IOs, 448049152 bytes 2023-12-10T19:06:27.119 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1200 IOs, 473930752 bytes 2023-12-10T19:06:27.297 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1300 IOs, 506416128 bytes 2023-12-10T19:06:27.471 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1400 IOs, 530195456 bytes 2023-12-10T19:06:27.701 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1500 IOs, 595031040 bytes 2023-12-10T19:06:27.814 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1600 IOs, 597462016 bytes 2023-12-10T19:06:28.082 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1700 IOs, 681985024 bytes 2023-12-10T19:06:28.224 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1800 IOs, 703159296 bytes 2023-12-10T19:06:28.357 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1900 IOs, 716426240 bytes 2023-12-10T19:06:28.511 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2000 IOs, 756387840 bytes 2023-12-10T19:06:28.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2100 IOs, 787966976 bytes 2023-12-10T19:06:28.914 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2200 IOs, 847715328 bytes 2023-12-10T19:06:29.023 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2300 IOs, 848342016 bytes 2023-12-10T19:06:29.122 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2400 IOs, 848977920 bytes 2023-12-10T19:06:29.302 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2500 IOs, 896120832 bytes 2023-12-10T19:06:29.472 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2600 IOs, 932226048 bytes 2023-12-10T19:06:29.601 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2700 IOs, 946981888 bytes 2023-12-10T19:06:29.726 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2800 IOs, 959535104 bytes 2023-12-10T19:06:29.740 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-12-10T19:06:29.740 INFO:teuthology.orchestra.run:waiting for 600 2023-12-10T19:06:29.740 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-12-10T19:06:29.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-10T19:06:29.741 DEBUG:teuthology.orchestra.run.smithi076:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-10T19:06:29.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-10T19:06:29.743 INFO:tasks.rbd_mirror_thrash:waiting for 108 secs before thrashing 2023-12-10T19:06:29.864 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2900 IOs, 969793536 bytes 2023-12-10T19:06:30.170 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3000 IOs, 1064683520 bytes 2023-12-10T19:06:30.327 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3100 IOs, 1090179072 bytes 2023-12-10T19:06:30.466 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3200 IOs, 1107718144 bytes 2023-12-10T19:06:30.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3300 IOs, 1165811712 bytes 2023-12-10T19:06:31.000 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3400 IOs, 1267209216 bytes 2023-12-10T19:06:31.184 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3500 IOs, 1304024064 bytes 2023-12-10T19:06:31.353 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3600 IOs, 1351329792 bytes 2023-12-10T19:06:31.578 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3700 IOs, 1401197568 bytes 2023-12-10T19:06:31.778 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3800 IOs, 1449528320 bytes 2023-12-10T19:06:31.995 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3900 IOs, 1506590720 bytes 2023-12-10T19:06:32.164 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4000 IOs, 1539128320 bytes 2023-12-10T19:06:32.377 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4100 IOs, 1587548160 bytes 2023-12-10T19:06:32.757 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4200 IOs, 1725363200 bytes 2023-12-10T19:06:32.896 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4300 IOs, 1758010368 bytes 2023-12-10T19:06:33.074 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4400 IOs, 1774993408 bytes 2023-12-10T19:06:33.243 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4500 IOs, 1803871232 bytes 2023-12-10T19:06:33.403 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4600 IOs, 1833138176 bytes 2023-12-10T19:06:33.608 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4700 IOs, 1880093696 bytes 2023-12-10T19:06:33.822 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4800 IOs, 1933545472 bytes 2023-12-10T19:06:34.031 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4900 IOs, 1984209920 bytes 2023-12-10T19:06:34.174 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5000 IOs, 2004633600 bytes 2023-12-10T19:06:34.308 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5100 IOs, 2017972224 bytes 2023-12-10T19:06:34.473 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5200 IOs, 2044969984 bytes 2023-12-10T19:06:34.669 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5300 IOs, 2103992320 bytes 2023-12-10T19:06:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5400 IOs, 2128233472 bytes 2023-12-10T19:06:35.117 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5500 IOs, 2207970304 bytes 2023-12-10T19:06:35.325 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5600 IOs, 2250370048 bytes 2023-12-10T19:06:35.558 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5700 IOs, 2323315712 bytes 2023-12-10T19:06:35.773 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5800 IOs, 2373726208 bytes 2023-12-10T19:06:35.904 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=124 2023-12-10T19:06:35.905 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T19:06:35.905 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 124 -eq 124 ] 2023-12-10T19:06:35.905 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:06:35.905 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_23 2023-12-10T19:06:35.905 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image_and_enable_mirror cluster2 mirror image_23 snapshot 128M 2023-12-10T19:06:35.905 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:06:35.905 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:06:35.906 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:06:35.906 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:06:35.906 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_23 2023-12-10T19:06:35.906 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:06:35.906 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:06:35.906 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n snapshot ] 2023-12-10T19:06:35.907 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:06:35.907 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image cluster2 mirror image_23 128M 2023-12-10T19:06:35.907 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:06:35.907 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:06:35.907 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:06:35.907 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:06:35.907 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_23 2023-12-10T19:06:35.907 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:06:35.907 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local size=128 2023-12-10T19:06:35.907 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n 128M ] 2023-12-10T19:06:35.907 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ size=128M 2023-12-10T19:06:35.907 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:06:35.907 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_23 2023-12-10T19:06:35.949 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ is_pool_mirror_mode_image mirror 2023-12-10T19:06:35.949 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:06:35.949 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ image = image ] 2023-12-10T19:06:35.949 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:06:35.949 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ enable_mirror cluster2 mirror image_23 snapshot 2023-12-10T19:06:35.950 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:06:35.950 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:06:35.950 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_23 2023-12-10T19:06:35.950 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:06:35.950 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_23 snapshot 2023-12-10T19:06:36.083 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirroring enabled 2023-12-10T19:06:36.091 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 info mirror/image_23 2023-12-10T19:06:36.137 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_23': 2023-12-10T19:06:36.137 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:06:36.137 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:06:36.137 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 1 2023-12-10T19:06:36.137 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 138459a304d0 2023-12-10T19:06:36.137 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.138459a304d0 2023-12-10T19:06:36.138 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:06:36.138 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering 2023-12-10T19:06:36.138 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:06:36.138 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:06:36.138 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:06:35 2023 2023-12-10T19:06:36.138 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:06:35 2023 2023-12-10T19:06:36.138 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:06:35 2023 2023-12-10T19:06:36.138 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:06:36.138 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:06:36.138 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: b1ff34f6-63f9-49d9-9315-4eabc97c393e 2023-12-10T19:06:36.138 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: true 2023-12-10T19:06:36.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T19:06:36.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stress_write_image cluster2 mirror image_23 2023-12-10T19:06:36.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:06:36.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:06:36.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_23 2023-12-10T19:06:36.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-10T19:06:36.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local duration=10 2023-12-10T19:06:36.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T19:06:36.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_23 --debug-rbd=20 --debug-journaler=20 2023-12-10T19:06:36.347 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:100 IOs, 35973120 bytes 2023-12-10T19:06:36.484 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:200 IOs, 55946240 bytes 2023-12-10T19:06:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:300 IOs, 72964096 bytes 2023-12-10T19:06:36.835 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:400 IOs, 129757184 bytes 2023-12-10T19:06:37.004 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr:2023-12-10T19:06:37.008+0000 7fa37264a640 -1 reset not still connected to 0x555757ba49c0 2023-12-10T19:06:37.015 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:500 IOs, 164583424 bytes 2023-12-10T19:06:37.340 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:600 IOs, 271859712 bytes 2023-12-10T19:06:37.522 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:700 IOs, 304920576 bytes 2023-12-10T19:06:37.662 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:800 IOs, 319920128 bytes 2023-12-10T19:06:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:900 IOs, 347869184 bytes 2023-12-10T19:06:38.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1000 IOs, 402254848 bytes 2023-12-10T19:06:38.205 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1100 IOs, 434008064 bytes 2023-12-10T19:06:38.447 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1200 IOs, 519658496 bytes 2023-12-10T19:06:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1300 IOs, 539236352 bytes 2023-12-10T19:06:38.833 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1400 IOs, 595919872 bytes 2023-12-10T19:06:38.984 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1500 IOs, 620763136 bytes 2023-12-10T19:06:39.167 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1600 IOs, 682084352 bytes 2023-12-10T19:06:39.383 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1700 IOs, 712317952 bytes 2023-12-10T19:06:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1800 IOs, 712928256 bytes 2023-12-10T19:06:39.693 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1900 IOs, 757158912 bytes 2023-12-10T19:06:39.856 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2000 IOs, 785043456 bytes 2023-12-10T19:06:40.049 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2100 IOs, 821749760 bytes 2023-12-10T19:06:40.205 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2200 IOs, 851022848 bytes 2023-12-10T19:06:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2300 IOs, 857381888 bytes 2023-12-10T19:06:40.441 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2400 IOs, 868400128 bytes 2023-12-10T19:06:40.739 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2500 IOs, 953729024 bytes 2023-12-10T19:06:40.923 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2600 IOs, 1008388096 bytes 2023-12-10T19:06:41.073 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2700 IOs, 1019062272 bytes 2023-12-10T19:06:41.263 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2800 IOs, 1062509568 bytes 2023-12-10T19:06:41.371 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2900 IOs, 1069382656 bytes 2023-12-10T19:06:41.555 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3000 IOs, 1106782208 bytes 2023-12-10T19:06:41.696 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3100 IOs, 1125953536 bytes 2023-12-10T19:06:41.838 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3200 IOs, 1147481088 bytes 2023-12-10T19:06:41.986 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3300 IOs, 1172874240 bytes 2023-12-10T19:06:42.223 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3400 IOs, 1229550592 bytes 2023-12-10T19:06:42.355 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3500 IOs, 1261481984 bytes 2023-12-10T19:06:42.640 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3600 IOs, 1334840320 bytes 2023-12-10T19:06:42.737 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.432888552233637) > weight (0.25) 2023-12-10T19:06:42.738 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.7265718965005443) > weight (0.25) 2023-12-10T19:06:42.738 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.8685769227770099) > weight (0.25) 2023-12-10T19:06:42.738 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.8688320247287558) > weight (0.25) 2023-12-10T19:06:42.738 INFO:tasks.rbd_mirror_thrash:waiting for 61 secs before thrashing 2023-12-10T19:06:42.885 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3700 IOs, 1409777664 bytes 2023-12-10T19:06:43.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3800 IOs, 1447638016 bytes 2023-12-10T19:06:43.219 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3900 IOs, 1463350272 bytes 2023-12-10T19:06:43.408 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4000 IOs, 1501258752 bytes 2023-12-10T19:06:43.594 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4100 IOs, 1545455616 bytes 2023-12-10T19:06:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4200 IOs, 1651776512 bytes 2023-12-10T19:06:44.108 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4300 IOs, 1683453952 bytes 2023-12-10T19:06:44.358 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4400 IOs, 1756770304 bytes 2023-12-10T19:06:44.548 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4500 IOs, 1792726016 bytes 2023-12-10T19:06:44.658 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4600 IOs, 1793954816 bytes 2023-12-10T19:06:44.873 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4700 IOs, 1850734592 bytes 2023-12-10T19:06:45.215 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4800 IOs, 1978220544 bytes 2023-12-10T19:06:45.427 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4900 IOs, 2020239360 bytes 2023-12-10T19:06:45.564 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5000 IOs, 2034540544 bytes 2023-12-10T19:06:45.739 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5100 IOs, 2070884352 bytes 2023-12-10T19:06:45.970 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5200 IOs, 2138873856 bytes 2023-12-10T19:06:46.129 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5300 IOs, 2161977344 bytes 2023-12-10T19:06:46.147 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=124 2023-12-10T19:06:46.147 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T19:06:46.147 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 124 -eq 124 ] 2023-12-10T19:06:46.147 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:06:46.147 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_24 2023-12-10T19:06:46.147 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image_and_enable_mirror cluster2 mirror image_24 snapshot 128M 2023-12-10T19:06:46.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:06:46.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:06:46.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:06:46.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:06:46.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_24 2023-12-10T19:06:46.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:06:46.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:06:46.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n snapshot ] 2023-12-10T19:06:46.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:06:46.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image cluster2 mirror image_24 128M 2023-12-10T19:06:46.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:06:46.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:06:46.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:06:46.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:06:46.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_24 2023-12-10T19:06:46.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:06:46.149 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local size=128 2023-12-10T19:06:46.149 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n 128M ] 2023-12-10T19:06:46.149 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ size=128M 2023-12-10T19:06:46.149 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:06:46.149 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_24 2023-12-10T19:06:46.193 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ is_pool_mirror_mode_image mirror 2023-12-10T19:06:46.194 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:06:46.194 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ image = image ] 2023-12-10T19:06:46.194 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:06:46.194 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ enable_mirror cluster2 mirror image_24 snapshot 2023-12-10T19:06:46.194 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:06:46.194 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:06:46.194 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_24 2023-12-10T19:06:46.194 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:06:46.194 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_24 snapshot 2023-12-10T19:06:46.331 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirroring enabled 2023-12-10T19:06:46.339 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 info mirror/image_24 2023-12-10T19:06:46.381 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_24': 2023-12-10T19:06:46.381 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:06:46.381 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:06:46.381 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 1 2023-12-10T19:06:46.381 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 13906ab5fec5 2023-12-10T19:06:46.381 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.13906ab5fec5 2023-12-10T19:06:46.381 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:06:46.381 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering 2023-12-10T19:06:46.381 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:06:46.381 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:06:46.381 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:06:46 2023 2023-12-10T19:06:46.381 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:06:46 2023 2023-12-10T19:06:46.382 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:06:46 2023 2023-12-10T19:06:46.382 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:06:46.382 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:06:46.382 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: 2920a7c1-c4ea-4349-87ad-88428404bcc1 2023-12-10T19:06:46.382 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: true 2023-12-10T19:06:46.385 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T19:06:46.385 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stress_write_image cluster2 mirror image_24 2023-12-10T19:06:46.385 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:06:46.385 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:06:46.385 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_24 2023-12-10T19:06:46.385 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-10T19:06:46.386 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local duration=14 2023-12-10T19:06:46.386 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T19:06:46.386 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_24 --debug-rbd=20 --debug-journaler=20 2023-12-10T19:06:46.633 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:100 IOs, 54884352 bytes 2023-12-10T19:06:46.837 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:200 IOs, 110422016 bytes 2023-12-10T19:06:46.956 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:300 IOs, 134594560 bytes 2023-12-10T19:06:47.150 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:400 IOs, 174482432 bytes 2023-12-10T19:06:47.359 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:500 IOs, 223102976 bytes 2023-12-10T19:06:47.517 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:600 IOs, 252312576 bytes 2023-12-10T19:06:47.695 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:700 IOs, 288762880 bytes 2023-12-10T19:06:47.927 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:800 IOs, 351748096 bytes 2023-12-10T19:06:48.106 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:900 IOs, 397880320 bytes 2023-12-10T19:06:48.726 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1000 IOs, 529044480 bytes 2023-12-10T19:06:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1100 IOs, 529666048 bytes 2023-12-10T19:06:48.997 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1200 IOs, 549340160 bytes 2023-12-10T19:06:49.144 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1300 IOs, 568317952 bytes 2023-12-10T19:06:49.351 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1400 IOs, 610623488 bytes 2023-12-10T19:06:49.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1500 IOs, 730907648 bytes 2023-12-10T19:06:49.899 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1600 IOs, 775828480 bytes 2023-12-10T19:06:50.016 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1700 IOs, 776450048 bytes 2023-12-10T19:06:50.192 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1800 IOs, 806819840 bytes 2023-12-10T19:06:50.419 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1900 IOs, 862140416 bytes 2023-12-10T19:06:50.565 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2000 IOs, 880013312 bytes 2023-12-10T19:06:50.691 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2100 IOs, 892208128 bytes 2023-12-10T19:06:50.834 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2200 IOs, 928901120 bytes 2023-12-10T19:06:50.996 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2300 IOs, 939660288 bytes 2023-12-10T19:06:51.180 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2400 IOs, 1006785536 bytes 2023-12-10T19:06:51.360 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2500 IOs, 1026074624 bytes 2023-12-10T19:06:51.595 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2600 IOs, 1085540352 bytes 2023-12-10T19:06:51.831 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2700 IOs, 1156400128 bytes 2023-12-10T19:06:52.084 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2800 IOs, 1222486016 bytes 2023-12-10T19:06:52.264 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2900 IOs, 1261807616 bytes 2023-12-10T19:06:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3000 IOs, 1333884928 bytes 2023-12-10T19:06:52.760 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3100 IOs, 1416137728 bytes 2023-12-10T19:06:52.904 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3200 IOs, 1434214400 bytes 2023-12-10T19:06:53.133 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3300 IOs, 1497824256 bytes 2023-12-10T19:06:53.286 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3400 IOs, 1547842560 bytes 2023-12-10T19:06:53.659 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3500 IOs, 1657729024 bytes 2023-12-10T19:06:53.774 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3600 IOs, 1660873728 bytes 2023-12-10T19:06:53.930 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3700 IOs, 1690506240 bytes 2023-12-10T19:06:54.179 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3800 IOs, 1750563840 bytes 2023-12-10T19:06:54.307 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3900 IOs, 1767851008 bytes 2023-12-10T19:06:54.487 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4000 IOs, 1806760960 bytes 2023-12-10T19:06:54.672 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4100 IOs, 1842038784 bytes 2023-12-10T19:06:54.908 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4200 IOs, 1918863360 bytes 2023-12-10T19:06:55.060 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4300 IOs, 1941755904 bytes 2023-12-10T19:06:55.216 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4400 IOs, 1961428992 bytes 2023-12-10T19:06:55.442 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4500 IOs, 2049944576 bytes 2023-12-10T19:06:55.634 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4600 IOs, 2068966400 bytes 2023-12-10T19:06:55.786 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4700 IOs, 2096420864 bytes 2023-12-10T19:06:55.983 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4800 IOs, 2144441344 bytes 2023-12-10T19:06:56.189 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4900 IOs, 2191142912 bytes 2023-12-10T19:06:56.328 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5000 IOs, 2203190272 bytes 2023-12-10T19:06:56.439 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5100 IOs, 2210489344 bytes 2023-12-10T19:06:56.561 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5200 IOs, 2211579904 bytes 2023-12-10T19:06:56.725 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5300 IOs, 2235981824 bytes 2023-12-10T19:06:56.934 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5400 IOs, 2283040768 bytes 2023-12-10T19:06:57.084 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5500 IOs, 2302714880 bytes 2023-12-10T19:06:57.273 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5600 IOs, 2343037952 bytes 2023-12-10T19:06:57.465 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5700 IOs, 2388600832 bytes 2023-12-10T19:06:57.605 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5800 IOs, 2401032192 bytes 2023-12-10T19:06:57.764 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5900 IOs, 2430843904 bytes 2023-12-10T19:06:57.921 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6000 IOs, 2471626752 bytes 2023-12-10T19:06:58.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6100 IOs, 2512519168 bytes 2023-12-10T19:06:58.336 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6200 IOs, 2549403648 bytes 2023-12-10T19:06:58.565 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6300 IOs, 2603767808 bytes 2023-12-10T19:06:58.710 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6400 IOs, 2623409152 bytes 2023-12-10T19:06:58.844 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6500 IOs, 2639577088 bytes 2023-12-10T19:06:59.046 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6600 IOs, 2685504512 bytes 2023-12-10T19:06:59.389 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6700 IOs, 2808842240 bytes 2023-12-10T19:06:59.623 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6800 IOs, 2862903296 bytes 2023-12-10T19:06:59.828 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6900 IOs, 2903282688 bytes 2023-12-10T19:07:00.019 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:7000 IOs, 2954136576 bytes 2023-12-10T19:07:00.229 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:7100 IOs, 2997629952 bytes 2023-12-10T19:07:00.392 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=124 2023-12-10T19:07:00.392 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T19:07:00.392 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 124 -eq 124 ] 2023-12-10T19:07:00.392 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:07:00.392 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_25 2023-12-10T19:07:00.392 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image_and_enable_mirror cluster2 mirror image_25 snapshot 128M 2023-12-10T19:07:00.392 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:07:00.392 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:07:00.392 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:07:00.393 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:07:00.393 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_25 2023-12-10T19:07:00.393 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:07:00.393 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:07:00.393 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n snapshot ] 2023-12-10T19:07:00.393 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:07:00.393 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image cluster2 mirror image_25 128M 2023-12-10T19:07:00.393 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:07:00.393 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:07:00.393 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:07:00.393 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:07:00.393 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_25 2023-12-10T19:07:00.393 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:07:00.393 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local size=128 2023-12-10T19:07:00.393 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n 128M ] 2023-12-10T19:07:00.394 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ size=128M 2023-12-10T19:07:00.394 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:07:00.394 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_25 2023-12-10T19:07:00.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ is_pool_mirror_mode_image mirror 2023-12-10T19:07:00.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:07:00.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ image = image ] 2023-12-10T19:07:00.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:07:00.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ enable_mirror cluster2 mirror image_25 snapshot 2023-12-10T19:07:00.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:07:00.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:07:00.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_25 2023-12-10T19:07:00.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:07:00.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_25 snapshot 2023-12-10T19:07:01.541 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirroring enabled 2023-12-10T19:07:01.552 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 info mirror/image_25 2023-12-10T19:07:01.598 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_25': 2023-12-10T19:07:01.599 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:07:01.599 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:07:01.599 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 1 2023-12-10T19:07:01.599 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 139c8ed81efe 2023-12-10T19:07:01.599 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.139c8ed81efe 2023-12-10T19:07:01.599 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:07:01.599 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering 2023-12-10T19:07:01.599 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:07:01.599 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:07:01.599 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:07:00 2023 2023-12-10T19:07:01.599 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:07:00 2023 2023-12-10T19:07:01.599 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:07:00 2023 2023-12-10T19:07:01.599 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:07:01.599 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:07:01.600 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: cd328d00-062e-4168-a2e8-c67acd6bae34 2023-12-10T19:07:01.600 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: true 2023-12-10T19:07:01.603 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T19:07:01.603 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stress_write_image cluster2 mirror image_25 2023-12-10T19:07:01.603 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:07:01.603 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:07:01.603 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_25 2023-12-10T19:07:01.603 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-10T19:07:01.604 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local duration=11 2023-12-10T19:07:01.605 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T19:07:01.605 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_25 --debug-rbd=20 --debug-journaler=20 2023-12-10T19:07:01.914 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:100 IOs, 72996864 bytes 2023-12-10T19:07:02.104 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:200 IOs, 114085888 bytes 2023-12-10T19:07:02.363 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:300 IOs, 184209408 bytes 2023-12-10T19:07:02.551 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:400 IOs, 211120128 bytes 2023-12-10T19:07:02.759 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:500 IOs, 255319040 bytes 2023-12-10T19:07:03.071 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:600 IOs, 359161856 bytes 2023-12-10T19:07:03.207 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:700 IOs, 376031232 bytes 2023-12-10T19:07:03.368 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:800 IOs, 398708736 bytes 2023-12-10T19:07:03.531 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:900 IOs, 417190912 bytes 2023-12-10T19:07:03.816 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1000 IOs, 506125312 bytes 2023-12-10T19:07:03.981 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1100 IOs, 540003328 bytes 2023-12-10T19:07:04.141 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1200 IOs, 559513600 bytes 2023-12-10T19:07:04.385 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1300 IOs, 630210560 bytes 2023-12-10T19:07:04.590 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1400 IOs, 667927552 bytes 2023-12-10T19:07:04.775 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1500 IOs, 713917440 bytes 2023-12-10T19:07:04.930 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1600 IOs, 741600256 bytes 2023-12-10T19:07:05.174 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1700 IOs, 810465280 bytes 2023-12-10T19:07:05.308 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1800 IOs, 831837184 bytes 2023-12-10T19:07:05.495 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1900 IOs, 861224960 bytes 2023-12-10T19:07:05.714 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2000 IOs, 919786496 bytes 2023-12-10T19:07:05.904 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2100 IOs, 948651008 bytes 2023-12-10T19:07:06.043 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2200 IOs, 967440384 bytes 2023-12-10T19:07:06.280 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2300 IOs, 1029919744 bytes 2023-12-10T19:07:06.409 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2400 IOs, 1060557824 bytes 2023-12-10T19:07:06.548 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2500 IOs, 1061153792 bytes 2023-12-10T19:07:06.747 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2600 IOs, 1100682240 bytes 2023-12-10T19:07:07.028 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2700 IOs, 1189277696 bytes 2023-12-10T19:07:07.242 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2800 IOs, 1239163904 bytes 2023-12-10T19:07:07.375 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2900 IOs, 1246187520 bytes 2023-12-10T19:07:07.544 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3000 IOs, 1297932288 bytes 2023-12-10T19:07:07.803 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3100 IOs, 1353657344 bytes 2023-12-10T19:07:07.964 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3200 IOs, 1378052096 bytes 2023-12-10T19:07:08.175 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3300 IOs, 1424794624 bytes 2023-12-10T19:07:08.445 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3400 IOs, 1512072192 bytes 2023-12-10T19:07:08.690 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3500 IOs, 1598353408 bytes 2023-12-10T19:07:09.007 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3600 IOs, 1689836544 bytes 2023-12-10T19:07:09.202 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3700 IOs, 1754806272 bytes 2023-12-10T19:07:09.464 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3800 IOs, 1829810176 bytes 2023-12-10T19:07:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3900 IOs, 1848892416 bytes 2023-12-10T19:07:10.016 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4000 IOs, 1970195456 bytes 2023-12-10T19:07:10.180 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4100 IOs, 2011216896 bytes 2023-12-10T19:07:10.397 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4200 IOs, 2052459520 bytes 2023-12-10T19:07:10.560 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4300 IOs, 2083191808 bytes 2023-12-10T19:07:10.811 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4400 IOs, 2157349888 bytes 2023-12-10T19:07:10.919 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4500 IOs, 2157954048 bytes 2023-12-10T19:07:11.142 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4600 IOs, 2209479680 bytes 2023-12-10T19:07:11.531 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4700 IOs, 2347152384 bytes 2023-12-10T19:07:11.811 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4800 IOs, 2428787712 bytes 2023-12-10T19:07:11.984 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4900 IOs, 2461681664 bytes 2023-12-10T19:07:12.104 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5000 IOs, 2475048960 bytes 2023-12-10T19:07:12.229 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5100 IOs, 2475677696 bytes 2023-12-10T19:07:12.408 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5200 IOs, 2531329024 bytes 2023-12-10T19:07:12.610 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=124 2023-12-10T19:07:12.610 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T19:07:12.610 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 124 -eq 124 ] 2023-12-10T19:07:12.610 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:07:12.610 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_26 2023-12-10T19:07:12.610 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image_and_enable_mirror cluster2 mirror image_26 snapshot 128M 2023-12-10T19:07:12.611 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:07:12.611 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:07:12.611 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:07:12.611 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:07:12.611 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_26 2023-12-10T19:07:12.611 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:07:12.611 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:07:12.611 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n snapshot ] 2023-12-10T19:07:12.611 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:07:12.611 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image cluster2 mirror image_26 128M 2023-12-10T19:07:12.611 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:07:12.611 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:07:12.611 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:07:12.611 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:07:12.611 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_26 2023-12-10T19:07:12.611 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:07:12.612 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local size=128 2023-12-10T19:07:12.612 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n 128M ] 2023-12-10T19:07:12.612 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ size=128M 2023-12-10T19:07:12.612 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:07:12.612 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_26 2023-12-10T19:07:12.661 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ is_pool_mirror_mode_image mirror 2023-12-10T19:07:12.662 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:07:12.662 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ image = image ] 2023-12-10T19:07:12.662 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:07:12.662 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ enable_mirror cluster2 mirror image_26 snapshot 2023-12-10T19:07:12.662 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:07:12.662 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:07:12.662 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_26 2023-12-10T19:07:12.662 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:07:12.663 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_26 snapshot 2023-12-10T19:07:13.580 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirroring enabled 2023-12-10T19:07:13.591 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 info mirror/image_26 2023-12-10T19:07:13.638 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_26': 2023-12-10T19:07:13.639 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:07:13.639 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:07:13.639 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 1 2023-12-10T19:07:13.639 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 13a8e96f403d 2023-12-10T19:07:13.639 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.13a8e96f403d 2023-12-10T19:07:13.639 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:07:13.639 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering 2023-12-10T19:07:13.639 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:07:13.639 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:07:13.639 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:07:12 2023 2023-12-10T19:07:13.639 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:07:12 2023 2023-12-10T19:07:13.639 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:07:12 2023 2023-12-10T19:07:13.639 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:07:13.639 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:07:13.640 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: b6b891c6-1bf3-4053-93b5-065fbe9ec26b 2023-12-10T19:07:13.640 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: true 2023-12-10T19:07:13.643 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T19:07:13.643 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stress_write_image cluster2 mirror image_26 2023-12-10T19:07:13.643 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:07:13.643 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:07:13.643 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_26 2023-12-10T19:07:13.643 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-10T19:07:13.644 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local duration=5 2023-12-10T19:07:13.644 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T19:07:13.644 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_26 --debug-rbd=20 --debug-journaler=20 2023-12-10T19:07:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:100 IOs, 15569920 bytes 2023-12-10T19:07:13.985 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:200 IOs, 55224320 bytes 2023-12-10T19:07:14.174 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:300 IOs, 95065088 bytes 2023-12-10T19:07:14.375 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:400 IOs, 147211264 bytes 2023-12-10T19:07:14.488 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:500 IOs, 173752320 bytes 2023-12-10T19:07:14.757 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:600 IOs, 236363776 bytes 2023-12-10T19:07:14.930 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:700 IOs, 271762432 bytes 2023-12-10T19:07:15.138 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:800 IOs, 319266816 bytes 2023-12-10T19:07:15.231 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:900 IOs, 319913984 bytes 2023-12-10T19:07:15.344 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1000 IOs, 320507904 bytes 2023-12-10T19:07:15.471 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1100 IOs, 333806592 bytes 2023-12-10T19:07:15.659 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1200 IOs, 374959104 bytes 2023-12-10T19:07:15.773 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1300 IOs, 383109120 bytes 2023-12-10T19:07:15.917 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1400 IOs, 402395136 bytes 2023-12-10T19:07:16.085 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1500 IOs, 431074304 bytes 2023-12-10T19:07:16.252 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1600 IOs, 462976000 bytes 2023-12-10T19:07:16.520 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1700 IOs, 537489408 bytes 2023-12-10T19:07:16.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1800 IOs, 639038464 bytes 2023-12-10T19:07:16.993 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1900 IOs, 675161088 bytes 2023-12-10T19:07:17.181 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2000 IOs, 735161344 bytes 2023-12-10T19:07:17.378 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2100 IOs, 771890176 bytes 2023-12-10T19:07:17.561 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2200 IOs, 796383232 bytes 2023-12-10T19:07:17.837 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2300 IOs, 906439680 bytes 2023-12-10T19:07:18.085 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2400 IOs, 946441216 bytes 2023-12-10T19:07:18.258 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2500 IOs, 981098496 bytes 2023-12-10T19:07:18.363 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2600 IOs, 981699584 bytes 2023-12-10T19:07:18.493 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2700 IOs, 994370560 bytes 2023-12-10T19:07:18.612 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2800 IOs, 1004104704 bytes 2023-12-10T19:07:18.649 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=124 2023-12-10T19:07:18.649 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T19:07:18.650 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 124 -eq 124 ] 2023-12-10T19:07:18.650 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:07:18.650 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_27 2023-12-10T19:07:18.650 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image_and_enable_mirror cluster2 mirror image_27 snapshot 128M 2023-12-10T19:07:18.650 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:07:18.650 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:07:18.650 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:07:18.650 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:07:18.650 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_27 2023-12-10T19:07:18.650 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:07:18.650 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:07:18.650 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n snapshot ] 2023-12-10T19:07:18.650 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:07:18.650 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image cluster2 mirror image_27 128M 2023-12-10T19:07:18.650 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:07:18.651 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:07:18.651 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:07:18.651 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:07:18.651 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_27 2023-12-10T19:07:18.651 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:07:18.651 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local size=128 2023-12-10T19:07:18.651 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n 128M ] 2023-12-10T19:07:18.651 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ size=128M 2023-12-10T19:07:18.651 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:07:18.651 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_27 2023-12-10T19:07:18.696 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ is_pool_mirror_mode_image mirror 2023-12-10T19:07:18.696 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:07:18.696 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ image = image ] 2023-12-10T19:07:18.696 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:07:18.696 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ enable_mirror cluster2 mirror image_27 snapshot 2023-12-10T19:07:18.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:07:18.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:07:18.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_27 2023-12-10T19:07:18.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:07:18.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_27 snapshot 2023-12-10T19:07:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirroring enabled 2023-12-10T19:07:19.640 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 info mirror/image_27 2023-12-10T19:07:19.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_27': 2023-12-10T19:07:19.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:07:19.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:07:19.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 1 2023-12-10T19:07:19.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 13b31cee4d3e 2023-12-10T19:07:19.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.13b31cee4d3e 2023-12-10T19:07:19.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:07:19.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering 2023-12-10T19:07:19.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:07:19.687 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:07:19.687 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:07:18 2023 2023-12-10T19:07:19.687 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:07:18 2023 2023-12-10T19:07:19.687 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:07:18 2023 2023-12-10T19:07:19.687 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:07:19.687 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:07:19.687 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: ef2fa6c2-fc2c-4d0b-9dde-4392f17b097a 2023-12-10T19:07:19.687 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: true 2023-12-10T19:07:19.690 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T19:07:19.690 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stress_write_image cluster2 mirror image_27 2023-12-10T19:07:19.690 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:07:19.690 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:07:19.690 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_27 2023-12-10T19:07:19.690 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-10T19:07:19.691 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local duration=10 2023-12-10T19:07:19.691 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T19:07:19.692 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_27 --debug-rbd=20 --debug-journaler=20 2023-12-10T19:07:19.843 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:100 IOs, 13797376 bytes 2023-12-10T19:07:20.019 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:200 IOs, 37722112 bytes 2023-12-10T19:07:20.378 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:300 IOs, 69735424 bytes 2023-12-10T19:07:20.545 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:400 IOs, 102109184 bytes 2023-12-10T19:07:20.794 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:500 IOs, 163696640 bytes 2023-12-10T19:07:20.958 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:600 IOs, 187438080 bytes 2023-12-10T19:07:21.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:700 IOs, 270921728 bytes 2023-12-10T19:07:21.411 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:800 IOs, 299564032 bytes 2023-12-10T19:07:21.576 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:900 IOs, 333117440 bytes 2023-12-10T19:07:21.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1000 IOs, 358050816 bytes 2023-12-10T19:07:21.885 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1100 IOs, 381708288 bytes 2023-12-10T19:07:22.068 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1200 IOs, 429812736 bytes 2023-12-10T19:07:22.275 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1300 IOs, 474029056 bytes 2023-12-10T19:07:22.426 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1400 IOs, 498550784 bytes 2023-12-10T19:07:22.616 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1500 IOs, 543901696 bytes 2023-12-10T19:07:22.741 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1600 IOs, 548238336 bytes 2023-12-10T19:07:22.887 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1700 IOs, 566506496 bytes 2023-12-10T19:07:23.019 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1800 IOs, 573329408 bytes 2023-12-10T19:07:23.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1900 IOs, 637143040 bytes 2023-12-10T19:07:23.443 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2000 IOs, 689285120 bytes 2023-12-10T19:07:23.610 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2100 IOs, 716875776 bytes 2023-12-10T19:07:23.902 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2200 IOs, 802273280 bytes 2023-12-10T19:07:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2300 IOs, 841374720 bytes 2023-12-10T19:07:24.353 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2400 IOs, 924294144 bytes 2023-12-10T19:07:24.464 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2500 IOs, 924910592 bytes 2023-12-10T19:07:24.691 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2600 IOs, 981739520 bytes 2023-12-10T19:07:24.805 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2700 IOs, 982342656 bytes 2023-12-10T19:07:24.953 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2800 IOs, 1002279936 bytes 2023-12-10T19:07:25.144 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2900 IOs, 1025241088 bytes 2023-12-10T19:07:25.379 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3000 IOs, 1089265664 bytes 2023-12-10T19:07:25.551 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3100 IOs, 1119770624 bytes 2023-12-10T19:07:25.699 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3200 IOs, 1146731520 bytes 2023-12-10T19:07:25.981 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3300 IOs, 1260588032 bytes 2023-12-10T19:07:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3400 IOs, 1288056832 bytes 2023-12-10T19:07:26.497 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3500 IOs, 1388561408 bytes 2023-12-10T19:07:26.672 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3600 IOs, 1421904896 bytes 2023-12-10T19:07:26.819 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3700 IOs, 1438879744 bytes 2023-12-10T19:07:26.950 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3800 IOs, 1458760704 bytes 2023-12-10T19:07:27.169 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3900 IOs, 1507260416 bytes 2023-12-10T19:07:27.496 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4000 IOs, 1597763584 bytes 2023-12-10T19:07:27.706 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4100 IOs, 1648357376 bytes 2023-12-10T19:07:27.904 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4200 IOs, 1700440064 bytes 2023-12-10T19:07:28.036 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4300 IOs, 1706000384 bytes 2023-12-10T19:07:28.165 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4400 IOs, 1712956416 bytes 2023-12-10T19:07:28.390 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4500 IOs, 1774098432 bytes 2023-12-10T19:07:28.582 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4600 IOs, 1817991168 bytes 2023-12-10T19:07:28.774 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4700 IOs, 1863597056 bytes 2023-12-10T19:07:28.955 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4800 IOs, 1899398144 bytes 2023-12-10T19:07:29.128 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4900 IOs, 1928754176 bytes 2023-12-10T19:07:29.406 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5000 IOs, 2011978752 bytes 2023-12-10T19:07:29.527 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5100 IOs, 2024714240 bytes 2023-12-10T19:07:29.696 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=124 2023-12-10T19:07:29.696 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T19:07:29.696 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 124 -eq 124 ] 2023-12-10T19:07:29.696 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:07:29.696 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_28 2023-12-10T19:07:29.696 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image_and_enable_mirror cluster2 mirror image_28 snapshot 128M 2023-12-10T19:07:29.696 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:07:29.696 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:07:29.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:07:29.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:07:29.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_28 2023-12-10T19:07:29.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:07:29.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:07:29.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n snapshot ] 2023-12-10T19:07:29.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:07:29.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image cluster2 mirror image_28 128M 2023-12-10T19:07:29.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:07:29.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:07:29.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:07:29.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:07:29.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_28 2023-12-10T19:07:29.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:07:29.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local size=128 2023-12-10T19:07:29.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n 128M ] 2023-12-10T19:07:29.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ size=128M 2023-12-10T19:07:29.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:07:29.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_28 2023-12-10T19:07:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ is_pool_mirror_mode_image mirror 2023-12-10T19:07:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:07:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ image = image ] 2023-12-10T19:07:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:07:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ enable_mirror cluster2 mirror image_28 snapshot 2023-12-10T19:07:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:07:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:07:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_28 2023-12-10T19:07:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:07:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_28 snapshot 2023-12-10T19:07:30.548 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirroring enabled 2023-12-10T19:07:30.560 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 info mirror/image_28 2023-12-10T19:07:30.607 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_28': 2023-12-10T19:07:30.607 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:07:30.607 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:07:30.607 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 1 2023-12-10T19:07:30.607 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 13bfa70a8c60 2023-12-10T19:07:30.607 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.13bfa70a8c60 2023-12-10T19:07:30.607 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:07:30.608 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering 2023-12-10T19:07:30.608 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:07:30.608 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:07:30.608 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:07:29 2023 2023-12-10T19:07:30.608 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:07:29 2023 2023-12-10T19:07:30.608 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:07:29 2023 2023-12-10T19:07:30.608 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:07:30.608 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:07:30.608 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: 8f8a9063-1808-4763-bb98-81cd23d63bba 2023-12-10T19:07:30.608 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: true 2023-12-10T19:07:30.611 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T19:07:30.611 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stress_write_image cluster2 mirror image_28 2023-12-10T19:07:30.611 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:07:30.611 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:07:30.611 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_28 2023-12-10T19:07:30.611 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-10T19:07:30.612 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local duration=9 2023-12-10T19:07:30.613 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T19:07:30.613 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_28 --debug-rbd=20 --debug-journaler=20 2023-12-10T19:07:30.848 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:100 IOs, 43362304 bytes 2023-12-10T19:07:31.060 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:200 IOs, 96822272 bytes 2023-12-10T19:07:31.307 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:300 IOs, 166709248 bytes 2023-12-10T19:07:31.416 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:400 IOs, 167664640 bytes 2023-12-10T19:07:31.568 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:500 IOs, 190662656 bytes 2023-12-10T19:07:31.766 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:600 IOs, 236984320 bytes 2023-12-10T19:07:31.991 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:700 IOs, 298706944 bytes 2023-12-10T19:07:32.149 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:800 IOs, 329169920 bytes 2023-12-10T19:07:32.385 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:900 IOs, 395186176 bytes 2023-12-10T19:07:32.586 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1000 IOs, 441939968 bytes 2023-12-10T19:07:32.733 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1100 IOs, 468102144 bytes 2023-12-10T19:07:32.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1200 IOs, 543762432 bytes 2023-12-10T19:07:33.187 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1300 IOs, 587077632 bytes 2023-12-10T19:07:33.370 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1400 IOs, 616878080 bytes 2023-12-10T19:07:33.533 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1500 IOs, 667752448 bytes 2023-12-10T19:07:33.726 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1600 IOs, 686511104 bytes 2023-12-10T19:07:34.063 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1700 IOs, 789198848 bytes 2023-12-10T19:07:34.203 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1800 IOs, 806124544 bytes 2023-12-10T19:07:34.360 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1900 IOs, 823988224 bytes 2023-12-10T19:07:34.556 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2000 IOs, 873224192 bytes 2023-12-10T19:07:34.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2100 IOs, 890412032 bytes 2023-12-10T19:07:34.864 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2200 IOs, 918242304 bytes 2023-12-10T19:07:35.060 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2300 IOs, 961651712 bytes 2023-12-10T19:07:35.265 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2400 IOs, 1011042304 bytes 2023-12-10T19:07:35.458 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2500 IOs, 1051115520 bytes 2023-12-10T19:07:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2600 IOs, 1058125824 bytes 2023-12-10T19:07:35.757 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2700 IOs, 1093588992 bytes 2023-12-10T19:07:35.914 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2800 IOs, 1107211264 bytes 2023-12-10T19:07:36.162 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2900 IOs, 1179799552 bytes 2023-12-10T19:07:36.385 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3000 IOs, 1235231744 bytes 2023-12-10T19:07:36.524 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3100 IOs, 1257075712 bytes 2023-12-10T19:07:36.734 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3200 IOs, 1311538176 bytes 2023-12-10T19:07:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3300 IOs, 1354808320 bytes 2023-12-10T19:07:37.056 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3400 IOs, 1383629824 bytes 2023-12-10T19:07:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3500 IOs, 1407810560 bytes 2023-12-10T19:07:37.390 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3600 IOs, 1437491200 bytes 2023-12-10T19:07:37.611 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3700 IOs, 1484246016 bytes 2023-12-10T19:07:37.759 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3800 IOs, 1503674368 bytes 2023-12-10T19:07:37.917 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3900 IOs, 1527493632 bytes 2023-12-10T19:07:38.066 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4000 IOs, 1549839360 bytes 2023-12-10T19:07:38.214 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4100 IOs, 1584661504 bytes 2023-12-10T19:07:38.354 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4200 IOs, 1585270784 bytes 2023-12-10T19:07:38.496 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4300 IOs, 1600210944 bytes 2023-12-10T19:07:38.679 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4400 IOs, 1638396928 bytes 2023-12-10T19:07:38.917 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4500 IOs, 1702208512 bytes 2023-12-10T19:07:39.136 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4600 IOs, 1773497344 bytes 2023-12-10T19:07:39.403 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4700 IOs, 1839591424 bytes 2023-12-10T19:07:39.572 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4800 IOs, 1843722240 bytes 2023-12-10T19:07:39.619 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=124 2023-12-10T19:07:39.619 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T19:07:39.619 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 124 -eq 124 ] 2023-12-10T19:07:39.620 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:07:39.620 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_29 2023-12-10T19:07:39.620 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image_and_enable_mirror cluster2 mirror image_29 snapshot 128M 2023-12-10T19:07:39.620 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:07:39.620 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:07:39.620 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:07:39.620 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:07:39.620 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_29 2023-12-10T19:07:39.620 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:07:39.620 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:07:39.620 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n snapshot ] 2023-12-10T19:07:39.620 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:07:39.620 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image cluster2 mirror image_29 128M 2023-12-10T19:07:39.620 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:07:39.620 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:07:39.621 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:07:39.621 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:07:39.621 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_29 2023-12-10T19:07:39.621 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:07:39.621 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local size=128 2023-12-10T19:07:39.621 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n 128M ] 2023-12-10T19:07:39.621 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ size=128M 2023-12-10T19:07:39.621 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:07:39.621 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_29 2023-12-10T19:07:39.664 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ is_pool_mirror_mode_image mirror 2023-12-10T19:07:39.664 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:07:39.664 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ image = image ] 2023-12-10T19:07:39.664 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:07:39.664 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ enable_mirror cluster2 mirror image_29 snapshot 2023-12-10T19:07:39.665 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:07:39.665 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:07:39.665 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_29 2023-12-10T19:07:39.665 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:07:39.665 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_29 snapshot 2023-12-10T19:07:40.535 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirroring enabled 2023-12-10T19:07:40.548 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 info mirror/image_29 2023-12-10T19:07:40.595 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_29': 2023-12-10T19:07:40.595 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:07:40.595 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:07:40.595 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 1 2023-12-10T19:07:40.595 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 13cbd52cbb32 2023-12-10T19:07:40.595 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.13cbd52cbb32 2023-12-10T19:07:40.596 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:07:40.596 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering 2023-12-10T19:07:40.596 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:07:40.596 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:07:40.596 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:07:39 2023 2023-12-10T19:07:40.596 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:07:39 2023 2023-12-10T19:07:40.596 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:07:39 2023 2023-12-10T19:07:40.596 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:07:40.596 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:07:40.596 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: dc1ce313-d80e-4cd6-a4dd-74802559481e 2023-12-10T19:07:40.596 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: true 2023-12-10T19:07:40.599 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T19:07:40.599 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stress_write_image cluster2 mirror image_29 2023-12-10T19:07:40.599 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:07:40.599 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:07:40.600 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_29 2023-12-10T19:07:40.600 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-10T19:07:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local duration=13 2023-12-10T19:07:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T19:07:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_29 --debug-rbd=20 --debug-journaler=20 2023-12-10T19:07:40.829 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:100 IOs, 39225344 bytes 2023-12-10T19:07:41.004 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:200 IOs, 71223296 bytes 2023-12-10T19:07:41.185 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:300 IOs, 109063168 bytes 2023-12-10T19:07:41.428 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:400 IOs, 175236096 bytes 2023-12-10T19:07:41.614 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:500 IOs, 211386368 bytes 2023-12-10T19:07:41.809 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:600 IOs, 277891072 bytes 2023-12-10T19:07:42.003 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:700 IOs, 314468352 bytes 2023-12-10T19:07:42.234 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:800 IOs, 371119104 bytes 2023-12-10T19:07:42.407 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:900 IOs, 401112064 bytes 2023-12-10T19:07:42.635 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1000 IOs, 453413888 bytes 2023-12-10T19:07:42.773 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1100 IOs, 464632832 bytes 2023-12-10T19:07:42.946 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1200 IOs, 491213824 bytes 2023-12-10T19:07:43.122 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1300 IOs, 515655680 bytes 2023-12-10T19:07:43.245 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1400 IOs, 523314176 bytes 2023-12-10T19:07:43.510 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1500 IOs, 600864768 bytes 2023-12-10T19:07:43.739 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2023-12-10T19:07:43.739 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2023-12-10T19:07:43.739 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2023-12-10T19:07:43.740 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2023-12-10T19:07:43.740 INFO:teuthology.orchestra.run:waiting for 600 2023-12-10T19:07:43.740 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi076.stderr:2023-12-10T19:07:43.746+0000 7ff632940640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 44797) UID: 1000 2023-12-10T19:07:43.863 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1600 IOs, 718883840 bytes 2023-12-10T19:07:44.088 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1700 IOs, 769124352 bytes 2023-12-10T19:07:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1800 IOs, 821516288 bytes 2023-12-10T19:07:44.454 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1900 IOs, 842525696 bytes 2023-12-10T19:07:44.659 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2000 IOs, 894408704 bytes 2023-12-10T19:07:44.807 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2100 IOs, 900715520 bytes 2023-12-10T19:07:44.921 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2200 IOs, 901350400 bytes 2023-12-10T19:07:45.162 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2300 IOs, 956507136 bytes 2023-12-10T19:07:45.303 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2400 IOs, 981483520 bytes 2023-12-10T19:07:45.490 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2500 IOs, 1005011968 bytes 2023-12-10T19:07:45.668 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2600 IOs, 1035538432 bytes 2023-12-10T19:07:45.952 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2700 IOs, 1110530048 bytes 2023-12-10T19:07:46.242 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2800 IOs, 1182467072 bytes 2023-12-10T19:07:46.495 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2900 IOs, 1254800384 bytes 2023-12-10T19:07:46.671 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3000 IOs, 1283865600 bytes 2023-12-10T19:07:46.839 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3100 IOs, 1312790528 bytes 2023-12-10T19:07:47.006 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3200 IOs, 1336372224 bytes 2023-12-10T19:07:47.163 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3300 IOs, 1376855040 bytes 2023-12-10T19:07:47.360 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3400 IOs, 1402175488 bytes 2023-12-10T19:07:47.539 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3500 IOs, 1431716864 bytes 2023-12-10T19:07:47.824 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3600 IOs, 1541192704 bytes 2023-12-10T19:07:48.005 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3700 IOs, 1547497472 bytes 2023-12-10T19:07:48.159 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3800 IOs, 1567694848 bytes 2023-12-10T19:07:48.509 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3900 IOs, 1687289856 bytes 2023-12-10T19:07:48.721 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4000 IOs, 1742043136 bytes 2023-12-10T19:07:48.958 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4100 IOs, 1791194112 bytes 2023-12-10T19:07:49.129 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4200 IOs, 1815588864 bytes 2023-12-10T19:07:49.515 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4300 IOs, 1945534464 bytes 2023-12-10T19:07:49.772 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4400 IOs, 2015308800 bytes 2023-12-10T19:07:49.842 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2023-12-10T19:07:49.842 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-12-10T19:07:49.842 DEBUG:teuthology.orchestra.run.smithi076:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-12-10T19:07:49.846 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-12-10T19:07:49.846 INFO:tasks.rbd_mirror_thrash:waiting for 75 secs before thrashing 2023-12-10T19:07:49.954 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4500 IOs, 2052118528 bytes 2023-12-10T19:07:50.214 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4600 IOs, 2106472448 bytes 2023-12-10T19:07:50.363 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4700 IOs, 2107079680 bytes 2023-12-10T19:07:50.503 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4800 IOs, 2116380672 bytes 2023-12-10T19:07:50.741 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4900 IOs, 2182582272 bytes 2023-12-10T19:07:50.962 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5000 IOs, 2234246144 bytes 2023-12-10T19:07:51.171 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5100 IOs, 2282077184 bytes 2023-12-10T19:07:51.328 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5200 IOs, 2304482304 bytes 2023-12-10T19:07:51.485 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5300 IOs, 2323696640 bytes 2023-12-10T19:07:51.605 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5400 IOs, 2330442752 bytes 2023-12-10T19:07:51.805 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5500 IOs, 2373554176 bytes 2023-12-10T19:07:51.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5600 IOs, 2410738688 bytes 2023-12-10T19:07:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5700 IOs, 2476293120 bytes 2023-12-10T19:07:52.470 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5800 IOs, 2531342336 bytes 2023-12-10T19:07:52.632 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5900 IOs, 2573358080 bytes 2023-12-10T19:07:52.937 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6000 IOs, 2661582848 bytes 2023-12-10T19:07:53.149 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6100 IOs, 2705657856 bytes 2023-12-10T19:07:53.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6200 IOs, 2739685376 bytes 2023-12-10T19:07:53.483 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6300 IOs, 2756139008 bytes 2023-12-10T19:07:53.607 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=124 2023-12-10T19:07:53.607 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T19:07:53.607 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 124 -eq 124 ] 2023-12-10T19:07:53.607 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:07:53.607 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_30 2023-12-10T19:07:53.608 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image_and_enable_mirror cluster2 mirror image_30 snapshot 128M 2023-12-10T19:07:53.608 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:07:53.608 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:07:53.608 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:07:53.608 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:07:53.609 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_30 2023-12-10T19:07:53.609 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:07:53.609 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:07:53.609 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n snapshot ] 2023-12-10T19:07:53.609 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:07:53.609 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image cluster2 mirror image_30 128M 2023-12-10T19:07:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:07:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:07:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:07:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:07:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_30 2023-12-10T19:07:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:07:53.611 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local size=128 2023-12-10T19:07:53.611 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n 128M ] 2023-12-10T19:07:53.611 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ size=128M 2023-12-10T19:07:53.611 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:07:53.611 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_30 2023-12-10T19:07:53.653 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ is_pool_mirror_mode_image mirror 2023-12-10T19:07:53.653 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:07:53.653 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ image = image ] 2023-12-10T19:07:53.653 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:07:53.654 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ enable_mirror cluster2 mirror image_30 snapshot 2023-12-10T19:07:53.654 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:07:53.654 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:07:53.654 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_30 2023-12-10T19:07:53.654 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:07:53.654 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_30 snapshot 2023-12-10T19:07:54.541 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirroring enabled 2023-12-10T19:07:54.554 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 info mirror/image_30 2023-12-10T19:07:54.599 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_30': 2023-12-10T19:07:54.599 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:07:54.599 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:07:54.599 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 1 2023-12-10T19:07:54.599 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 13e11cd7423a 2023-12-10T19:07:54.599 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.13e11cd7423a 2023-12-10T19:07:54.599 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:07:54.599 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering 2023-12-10T19:07:54.599 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:07:54.599 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:07:54.600 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:07:53 2023 2023-12-10T19:07:54.600 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:07:53 2023 2023-12-10T19:07:54.600 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:07:53 2023 2023-12-10T19:07:54.600 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:07:54.600 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:07:54.600 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: 4d4eb965-6ed2-4ccb-a76d-8e89ea006260 2023-12-10T19:07:54.600 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: true 2023-12-10T19:07:54.603 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T19:07:54.603 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stress_write_image cluster2 mirror image_30 2023-12-10T19:07:54.603 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:07:54.603 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:07:54.603 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_30 2023-12-10T19:07:54.603 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-10T19:07:54.604 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local duration=10 2023-12-10T19:07:54.605 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T19:07:54.605 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_30 --debug-rbd=20 --debug-journaler=20 2023-12-10T19:07:54.887 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:100 IOs, 70961152 bytes 2023-12-10T19:07:55.047 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:200 IOs, 104144896 bytes 2023-12-10T19:07:55.209 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:300 IOs, 134797312 bytes 2023-12-10T19:07:55.366 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:400 IOs, 173926400 bytes 2023-12-10T19:07:55.516 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:500 IOs, 185690112 bytes 2023-12-10T19:07:55.648 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:600 IOs, 205833216 bytes 2023-12-10T19:07:55.832 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:700 IOs, 251399168 bytes 2023-12-10T19:07:56.016 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:800 IOs, 285834240 bytes 2023-12-10T19:07:56.145 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:900 IOs, 292982784 bytes 2023-12-10T19:07:56.358 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1000 IOs, 343077888 bytes 2023-12-10T19:07:56.625 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1100 IOs, 432616448 bytes 2023-12-10T19:07:56.811 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1200 IOs, 458619904 bytes 2023-12-10T19:07:56.980 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1300 IOs, 492127232 bytes 2023-12-10T19:07:57.174 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1400 IOs, 532162560 bytes 2023-12-10T19:07:57.404 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1500 IOs, 591232000 bytes 2023-12-10T19:07:57.514 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1600 IOs, 594904064 bytes 2023-12-10T19:07:57.687 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1700 IOs, 624435200 bytes 2023-12-10T19:07:57.855 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1800 IOs, 660537344 bytes 2023-12-10T19:07:58.069 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1900 IOs, 702257152 bytes 2023-12-10T19:07:58.228 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2000 IOs, 719656960 bytes 2023-12-10T19:07:58.477 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2100 IOs, 788396032 bytes 2023-12-10T19:07:58.606 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2200 IOs, 798605312 bytes 2023-12-10T19:07:58.753 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2300 IOs, 820512768 bytes 2023-12-10T19:07:58.873 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2400 IOs, 821144576 bytes 2023-12-10T19:07:59.075 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2500 IOs, 867881984 bytes 2023-12-10T19:07:59.186 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2600 IOs, 868478976 bytes 2023-12-10T19:07:59.460 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2700 IOs, 968889344 bytes 2023-12-10T19:07:59.690 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2800 IOs, 1035415552 bytes 2023-12-10T19:07:59.955 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2900 IOs, 1090920448 bytes 2023-12-10T19:08:00.166 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3000 IOs, 1162978304 bytes 2023-12-10T19:08:00.380 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3100 IOs, 1197850624 bytes 2023-12-10T19:08:00.502 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3200 IOs, 1198452736 bytes 2023-12-10T19:08:00.657 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3300 IOs, 1218123776 bytes 2023-12-10T19:08:00.841 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3400 IOs, 1274498048 bytes 2023-12-10T19:08:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3500 IOs, 1339608064 bytes 2023-12-10T19:08:01.279 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3600 IOs, 1392037888 bytes 2023-12-10T19:08:01.590 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3700 IOs, 1462662144 bytes 2023-12-10T19:08:01.822 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3800 IOs, 1532667904 bytes 2023-12-10T19:08:01.993 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3900 IOs, 1555767296 bytes 2023-12-10T19:08:02.350 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4000 IOs, 1679260672 bytes 2023-12-10T19:08:02.523 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4100 IOs, 1705743360 bytes 2023-12-10T19:08:02.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4200 IOs, 1740498944 bytes 2023-12-10T19:08:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4300 IOs, 1782461440 bytes 2023-12-10T19:08:03.122 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4400 IOs, 1839755264 bytes 2023-12-10T19:08:03.338 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4500 IOs, 1884089344 bytes 2023-12-10T19:08:03.545 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4600 IOs, 1931102208 bytes 2023-12-10T19:08:03.765 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4700 IOs, 1987203072 bytes 2023-12-10T19:08:03.919 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4800 IOs, 2008297472 bytes 2023-12-10T19:08:04.147 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4900 IOs, 2067236864 bytes 2023-12-10T19:08:04.438 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5000 IOs, 2157684736 bytes 2023-12-10T19:08:04.610 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=124 2023-12-10T19:08:04.610 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T19:08:04.611 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 124 -eq 124 ] 2023-12-10T19:08:04.611 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:08:04.611 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_31 2023-12-10T19:08:04.611 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image_and_enable_mirror cluster2 mirror image_31 snapshot 128M 2023-12-10T19:08:04.611 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:08:04.611 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:08:04.611 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:08:04.611 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:08:04.611 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_31 2023-12-10T19:08:04.611 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:08:04.611 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:08:04.611 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n snapshot ] 2023-12-10T19:08:04.611 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:08:04.611 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image cluster2 mirror image_31 128M 2023-12-10T19:08:04.611 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:08:04.612 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:08:04.612 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:08:04.612 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:08:04.612 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_31 2023-12-10T19:08:04.612 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:08:04.612 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local size=128 2023-12-10T19:08:04.612 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n 128M ] 2023-12-10T19:08:04.612 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ size=128M 2023-12-10T19:08:04.612 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:08:04.612 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_31 2023-12-10T19:08:04.659 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ is_pool_mirror_mode_image mirror 2023-12-10T19:08:04.659 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:08:04.659 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ image = image ] 2023-12-10T19:08:04.659 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:08:04.659 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ enable_mirror cluster2 mirror image_31 snapshot 2023-12-10T19:08:04.659 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:08:04.659 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:08:04.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_31 2023-12-10T19:08:04.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:08:04.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_31 snapshot 2023-12-10T19:08:05.061 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirroring enabled 2023-12-10T19:08:05.070 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 info mirror/image_31 2023-12-10T19:08:05.115 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_31': 2023-12-10T19:08:05.115 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:08:05.115 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:08:05.115 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 1 2023-12-10T19:08:05.115 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 13edfb375abc 2023-12-10T19:08:05.115 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.13edfb375abc 2023-12-10T19:08:05.115 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:08:05.115 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering 2023-12-10T19:08:05.115 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:08:05.115 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:08:05.116 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:08:04 2023 2023-12-10T19:08:05.116 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:08:04 2023 2023-12-10T19:08:05.116 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:08:04 2023 2023-12-10T19:08:05.116 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:08:05.116 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:08:05.116 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: 899d2f20-51d9-4c49-ba24-b07293c27322 2023-12-10T19:08:05.116 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: true 2023-12-10T19:08:05.119 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T19:08:05.119 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stress_write_image cluster2 mirror image_31 2023-12-10T19:08:05.119 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:08:05.119 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:08:05.119 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_31 2023-12-10T19:08:05.119 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-10T19:08:05.120 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local duration=7 2023-12-10T19:08:05.120 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T19:08:05.120 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_31 --debug-rbd=20 --debug-journaler=20 2023-12-10T19:08:05.328 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:100 IOs, 28140544 bytes 2023-12-10T19:08:05.474 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:200 IOs, 54758400 bytes 2023-12-10T19:08:05.652 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:300 IOs, 90824704 bytes 2023-12-10T19:08:05.779 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:400 IOs, 116406272 bytes 2023-12-10T19:08:06.044 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:500 IOs, 199711744 bytes 2023-12-10T19:08:06.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:600 IOs, 265734144 bytes 2023-12-10T19:08:06.474 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:700 IOs, 279136256 bytes 2023-12-10T19:08:06.670 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:800 IOs, 321865728 bytes 2023-12-10T19:08:06.896 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:900 IOs, 382189568 bytes 2023-12-10T19:08:07.059 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1000 IOs, 408599552 bytes 2023-12-10T19:08:07.308 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1100 IOs, 491015168 bytes 2023-12-10T19:08:07.580 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1200 IOs, 538048512 bytes 2023-12-10T19:08:07.750 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1300 IOs, 566312960 bytes 2023-12-10T19:08:07.952 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1400 IOs, 610756608 bytes 2023-12-10T19:08:08.100 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1500 IOs, 628691968 bytes 2023-12-10T19:08:08.273 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1600 IOs, 655620096 bytes 2023-12-10T19:08:08.529 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1700 IOs, 722366464 bytes 2023-12-10T19:08:08.920 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1800 IOs, 777355264 bytes 2023-12-10T19:08:09.067 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1900 IOs, 777964544 bytes 2023-12-10T19:08:09.287 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2000 IOs, 815122432 bytes 2023-12-10T19:08:09.476 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2100 IOs, 882128896 bytes 2023-12-10T19:08:09.687 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2200 IOs, 905859072 bytes 2023-12-10T19:08:09.948 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2300 IOs, 993329152 bytes 2023-12-10T19:08:10.086 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2400 IOs, 997226496 bytes 2023-12-10T19:08:10.363 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2500 IOs, 1075614720 bytes 2023-12-10T19:08:10.596 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2600 IOs, 1140149248 bytes 2023-12-10T19:08:10.786 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2700 IOs, 1185744896 bytes 2023-12-10T19:08:10.997 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2800 IOs, 1244532736 bytes 2023-12-10T19:08:11.177 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2900 IOs, 1281798144 bytes 2023-12-10T19:08:11.512 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3000 IOs, 1391133696 bytes 2023-12-10T19:08:11.724 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3100 IOs, 1448227840 bytes 2023-12-10T19:08:11.866 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3200 IOs, 1466334208 bytes 2023-12-10T19:08:12.001 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3300 IOs, 1479452672 bytes 2023-12-10T19:08:12.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=124 2023-12-10T19:08:12.125 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T19:08:12.125 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 124 -eq 124 ] 2023-12-10T19:08:12.125 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:08:12.125 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_32 2023-12-10T19:08:12.125 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image_and_enable_mirror cluster2 mirror image_32 snapshot 128M 2023-12-10T19:08:12.125 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:08:12.125 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:08:12.125 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:08:12.125 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:08:12.125 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_32 2023-12-10T19:08:12.125 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:08:12.125 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:08:12.125 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n snapshot ] 2023-12-10T19:08:12.125 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:08:12.125 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image cluster2 mirror image_32 128M 2023-12-10T19:08:12.126 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:08:12.126 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:08:12.126 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:08:12.126 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:08:12.126 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_32 2023-12-10T19:08:12.126 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:08:12.126 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local size=128 2023-12-10T19:08:12.126 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n 128M ] 2023-12-10T19:08:12.126 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ size=128M 2023-12-10T19:08:12.126 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:08:12.126 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_32 2023-12-10T19:08:12.173 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ is_pool_mirror_mode_image mirror 2023-12-10T19:08:12.174 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:08:12.174 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ image = image ] 2023-12-10T19:08:12.174 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:08:12.174 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ enable_mirror cluster2 mirror image_32 snapshot 2023-12-10T19:08:12.174 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:08:12.174 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:08:12.174 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_32 2023-12-10T19:08:12.174 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:08:12.174 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_32 snapshot 2023-12-10T19:08:12.576 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirroring enabled 2023-12-10T19:08:12.587 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 info mirror/image_32 2023-12-10T19:08:12.635 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_32': 2023-12-10T19:08:12.635 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:08:12.635 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:08:12.635 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 1 2023-12-10T19:08:12.635 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 13f9eb6f93e5 2023-12-10T19:08:12.636 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.13f9eb6f93e5 2023-12-10T19:08:12.636 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:08:12.636 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering 2023-12-10T19:08:12.636 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:08:12.636 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:08:12.636 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:08:12 2023 2023-12-10T19:08:12.636 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:08:12 2023 2023-12-10T19:08:12.636 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:08:12 2023 2023-12-10T19:08:12.636 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:08:12.636 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:08:12.636 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: f5919ca9-8590-41d3-8074-153a93205199 2023-12-10T19:08:12.636 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: true 2023-12-10T19:08:12.639 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T19:08:12.639 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stress_write_image cluster2 mirror image_32 2023-12-10T19:08:12.639 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:08:12.639 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:08:12.640 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_32 2023-12-10T19:08:12.640 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-10T19:08:12.641 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local duration=8 2023-12-10T19:08:12.641 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T19:08:12.641 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_32 --debug-rbd=20 --debug-journaler=20 2023-12-10T19:08:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:100 IOs, 63953920 bytes 2023-12-10T19:08:13.118 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:200 IOs, 151706624 bytes 2023-12-10T19:08:13.391 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:300 IOs, 205361152 bytes 2023-12-10T19:08:13.623 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:400 IOs, 305235968 bytes 2023-12-10T19:08:13.852 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:500 IOs, 326808576 bytes 2023-12-10T19:08:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:600 IOs, 348996608 bytes 2023-12-10T19:08:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:700 IOs, 384259072 bytes 2023-12-10T19:08:14.330 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:800 IOs, 425635840 bytes 2023-12-10T19:08:14.586 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:900 IOs, 478840832 bytes 2023-12-10T19:08:14.746 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1000 IOs, 522921984 bytes 2023-12-10T19:08:14.979 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1100 IOs, 572566528 bytes 2023-12-10T19:08:15.098 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1200 IOs, 574166016 bytes 2023-12-10T19:08:15.288 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1300 IOs, 613899264 bytes 2023-12-10T19:08:15.426 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1400 IOs, 626657280 bytes 2023-12-10T19:08:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1500 IOs, 675297280 bytes 2023-12-10T19:08:15.777 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1600 IOs, 695651328 bytes 2023-12-10T19:08:16.134 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1700 IOs, 842915840 bytes 2023-12-10T19:08:16.342 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1800 IOs, 880963584 bytes 2023-12-10T19:08:16.547 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1900 IOs, 944523264 bytes 2023-12-10T19:08:16.809 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2000 IOs, 1015646208 bytes 2023-12-10T19:08:17.004 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2100 IOs, 1079442432 bytes 2023-12-10T19:08:17.322 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2200 IOs, 1166520320 bytes 2023-12-10T19:08:17.468 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2300 IOs, 1198590976 bytes 2023-12-10T19:08:17.687 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2400 IOs, 1246635008 bytes 2023-12-10T19:08:17.745 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-12-10T19:08:17.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-12-10T19:08:17.745 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2023-12-10T19:08:17.745 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-12-10T19:08:17.745 INFO:teuthology.orchestra.run:waiting for 600 2023-12-10T19:08:17.746 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr:2023-12-10T19:08:17.753+0000 7fe08381c640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 50460) UID: 1000 2023-12-10T19:08:17.844 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2500 IOs, 1264566272 bytes 2023-12-10T19:08:18.130 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2600 IOs, 1349400576 bytes 2023-12-10T19:08:18.344 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2700 IOs, 1403843584 bytes 2023-12-10T19:08:18.584 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2800 IOs, 1470020608 bytes 2023-12-10T19:08:18.749 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2900 IOs, 1501126656 bytes 2023-12-10T19:08:18.918 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3000 IOs, 1531155456 bytes 2023-12-10T19:08:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3100 IOs, 1547274240 bytes 2023-12-10T19:08:19.228 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3200 IOs, 1585178624 bytes 2023-12-10T19:08:19.377 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3300 IOs, 1605302272 bytes 2023-12-10T19:08:19.548 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3400 IOs, 1629657088 bytes 2023-12-10T19:08:19.709 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3500 IOs, 1662559232 bytes 2023-12-10T19:08:20.041 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3600 IOs, 1767042048 bytes 2023-12-10T19:08:20.198 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3700 IOs, 1791408128 bytes 2023-12-10T19:08:20.353 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3800 IOs, 1822926848 bytes 2023-12-10T19:08:20.558 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3900 IOs, 1868550144 bytes 2023-12-10T19:08:20.647 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=124 2023-12-10T19:08:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T19:08:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 124 -eq 124 ] 2023-12-10T19:08:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:08:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_33 2023-12-10T19:08:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image_and_enable_mirror cluster2 mirror image_33 snapshot 128M 2023-12-10T19:08:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:08:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:08:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:08:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:08:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_33 2023-12-10T19:08:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:08:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:08:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n snapshot ] 2023-12-10T19:08:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:08:20.649 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image cluster2 mirror image_33 128M 2023-12-10T19:08:20.649 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:08:20.649 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:08:20.649 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:08:20.649 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:08:20.649 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_33 2023-12-10T19:08:20.649 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:08:20.649 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local size=128 2023-12-10T19:08:20.649 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n 128M ] 2023-12-10T19:08:20.649 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ size=128M 2023-12-10T19:08:20.649 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:08:20.649 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_33 2023-12-10T19:08:20.694 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ is_pool_mirror_mode_image mirror 2023-12-10T19:08:20.695 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:08:20.695 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ image = image ] 2023-12-10T19:08:20.695 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:08:20.695 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ enable_mirror cluster2 mirror image_33 snapshot 2023-12-10T19:08:20.695 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:08:20.695 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:08:20.695 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_33 2023-12-10T19:08:20.695 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:08:20.695 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_33 snapshot 2023-12-10T19:08:21.609 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirroring enabled 2023-12-10T19:08:21.621 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 info mirror/image_33 2023-12-10T19:08:21.668 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_33': 2023-12-10T19:08:21.668 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:08:21.668 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:08:21.669 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 1 2023-12-10T19:08:21.669 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 1405b2648769 2023-12-10T19:08:21.669 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.1405b2648769 2023-12-10T19:08:21.669 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:08:21.669 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering 2023-12-10T19:08:21.669 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:08:21.669 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:08:21.669 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:08:20 2023 2023-12-10T19:08:21.669 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:08:20 2023 2023-12-10T19:08:21.669 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:08:20 2023 2023-12-10T19:08:21.669 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:08:21.669 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:08:21.669 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: 195ff7a6-d9ef-4fb6-afac-e87b92563048 2023-12-10T19:08:21.669 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: true 2023-12-10T19:08:21.673 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T19:08:21.673 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stress_write_image cluster2 mirror image_33 2023-12-10T19:08:21.673 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:08:21.673 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:08:21.673 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_33 2023-12-10T19:08:21.673 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-10T19:08:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local duration=11 2023-12-10T19:08:21.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T19:08:21.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_33 --debug-rbd=20 --debug-journaler=20 2023-12-10T19:08:21.969 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:100 IOs, 75489280 bytes 2023-12-10T19:08:22.223 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:200 IOs, 156738560 bytes 2023-12-10T19:08:22.421 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:300 IOs, 200011776 bytes 2023-12-10T19:08:22.719 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:400 IOs, 284890112 bytes 2023-12-10T19:08:23.073 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:500 IOs, 401733632 bytes 2023-12-10T19:08:23.307 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:600 IOs, 462051328 bytes 2023-12-10T19:08:23.536 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:700 IOs, 528952320 bytes 2023-12-10T19:08:23.672 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:800 IOs, 535375872 bytes 2023-12-10T19:08:23.793 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:900 IOs, 542257152 bytes 2023-12-10T19:08:23.848 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-12-10T19:08:23.848 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-10T19:08:23.848 DEBUG:teuthology.orchestra.run.smithi076:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-10T19:08:23.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-10T19:08:23.851 INFO:tasks.rbd_mirror_thrash:waiting for 73 secs before thrashing 2023-12-10T19:08:24.015 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1000 IOs, 625005568 bytes 2023-12-10T19:08:24.198 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1100 IOs, 670227456 bytes 2023-12-10T19:08:24.397 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1200 IOs, 682415104 bytes 2023-12-10T19:08:24.541 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1300 IOs, 699162624 bytes 2023-12-10T19:08:24.685 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1400 IOs, 719672320 bytes 2023-12-10T19:08:24.955 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1500 IOs, 795515904 bytes 2023-12-10T19:08:25.073 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1600 IOs, 796125184 bytes 2023-12-10T19:08:25.228 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1700 IOs, 821776384 bytes 2023-12-10T19:08:25.337 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1800 IOs, 823689216 bytes 2023-12-10T19:08:25.455 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1900 IOs, 829190144 bytes 2023-12-10T19:08:25.658 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2000 IOs, 875053056 bytes 2023-12-10T19:08:25.806 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2100 IOs, 900039680 bytes 2023-12-10T19:08:25.924 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2200 IOs, 902683648 bytes 2023-12-10T19:08:26.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2300 IOs, 925964288 bytes 2023-12-10T19:08:26.231 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2400 IOs, 947910656 bytes 2023-12-10T19:08:26.433 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2500 IOs, 987770880 bytes 2023-12-10T19:08:26.605 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2600 IOs, 1016203264 bytes 2023-12-10T19:08:26.791 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2700 IOs, 1053171712 bytes 2023-12-10T19:08:26.978 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2800 IOs, 1093017600 bytes 2023-12-10T19:08:27.118 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2900 IOs, 1108603904 bytes 2023-12-10T19:08:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3000 IOs, 1144143872 bytes 2023-12-10T19:08:27.553 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3100 IOs, 1220145152 bytes 2023-12-10T19:08:27.793 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3200 IOs, 1289877504 bytes 2023-12-10T19:08:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3300 IOs, 1314231296 bytes 2023-12-10T19:08:28.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3400 IOs, 1344885760 bytes 2023-12-10T19:08:28.266 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3500 IOs, 1351601152 bytes 2023-12-10T19:08:28.485 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3600 IOs, 1410502656 bytes 2023-12-10T19:08:28.683 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3700 IOs, 1447724032 bytes 2023-12-10T19:08:28.953 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3800 IOs, 1533344768 bytes 2023-12-10T19:08:29.157 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3900 IOs, 1566031872 bytes 2023-12-10T19:08:29.311 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4000 IOs, 1587792896 bytes 2023-12-10T19:08:29.527 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4100 IOs, 1644944384 bytes 2023-12-10T19:08:29.747 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4200 IOs, 1706795008 bytes 2023-12-10T19:08:29.917 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4300 IOs, 1741191168 bytes 2023-12-10T19:08:30.064 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4400 IOs, 1759293440 bytes 2023-12-10T19:08:30.179 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4500 IOs, 1763439616 bytes 2023-12-10T19:08:30.404 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4600 IOs, 1814141952 bytes 2023-12-10T19:08:30.585 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4700 IOs, 1856266240 bytes 2023-12-10T19:08:30.879 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4800 IOs, 1962441728 bytes 2023-12-10T19:08:31.033 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4900 IOs, 1985092608 bytes 2023-12-10T19:08:31.167 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5000 IOs, 1998099456 bytes 2023-12-10T19:08:31.397 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5100 IOs, 2054839296 bytes 2023-12-10T19:08:31.576 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5200 IOs, 2093032448 bytes 2023-12-10T19:08:31.724 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5300 IOs, 2109797376 bytes 2023-12-10T19:08:31.945 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5400 IOs, 2168125440 bytes 2023-12-10T19:08:32.167 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5500 IOs, 2222041088 bytes 2023-12-10T19:08:32.375 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5600 IOs, 2267415552 bytes 2023-12-10T19:08:32.589 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5700 IOs, 2322540544 bytes 2023-12-10T19:08:32.682 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=124 2023-12-10T19:08:32.682 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T19:08:32.682 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 124 -eq 124 ] 2023-12-10T19:08:32.683 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:08:32.683 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_34 2023-12-10T19:08:32.683 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image_and_enable_mirror cluster2 mirror image_34 snapshot 128M 2023-12-10T19:08:32.683 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:08:32.683 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:08:32.683 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:08:32.683 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:08:32.683 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_34 2023-12-10T19:08:32.683 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:08:32.683 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:08:32.683 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n snapshot ] 2023-12-10T19:08:32.683 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:08:32.683 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image cluster2 mirror image_34 128M 2023-12-10T19:08:32.683 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:08:32.684 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:08:32.684 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:08:32.684 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:08:32.684 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_34 2023-12-10T19:08:32.684 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:08:32.684 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local size=128 2023-12-10T19:08:32.684 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n 128M ] 2023-12-10T19:08:32.684 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ size=128M 2023-12-10T19:08:32.684 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:08:32.684 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_34 2023-12-10T19:08:32.729 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ is_pool_mirror_mode_image mirror 2023-12-10T19:08:32.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:08:32.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ image = image ] 2023-12-10T19:08:32.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:08:32.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ enable_mirror cluster2 mirror image_34 snapshot 2023-12-10T19:08:32.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:08:32.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:08:32.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_34 2023-12-10T19:08:32.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:08:32.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_34 snapshot 2023-12-10T19:08:33.641 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirroring enabled 2023-12-10T19:08:33.652 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 info mirror/image_34 2023-12-10T19:08:33.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_34': 2023-12-10T19:08:33.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:08:33.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:08:33.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 1 2023-12-10T19:08:33.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 141536efcd05 2023-12-10T19:08:33.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.141536efcd05 2023-12-10T19:08:33.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:08:33.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering 2023-12-10T19:08:33.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:08:33.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:08:33.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:08:32 2023 2023-12-10T19:08:33.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:08:32 2023 2023-12-10T19:08:33.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:08:32 2023 2023-12-10T19:08:33.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:08:33.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:08:33.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: d25b8883-1661-4a4c-bc7a-8d186c1f1d26 2023-12-10T19:08:33.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: true 2023-12-10T19:08:33.701 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T19:08:33.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stress_write_image cluster2 mirror image_34 2023-12-10T19:08:33.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:08:33.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:08:33.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_34 2023-12-10T19:08:33.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-10T19:08:33.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local duration=12 2023-12-10T19:08:33.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T19:08:33.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_34 --debug-rbd=20 --debug-journaler=20 2023-12-10T19:08:33.878 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:100 IOs, 10033152 bytes 2023-12-10T19:08:34.041 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:200 IOs, 40423424 bytes 2023-12-10T19:08:34.353 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:300 IOs, 190135296 bytes 2023-12-10T19:08:34.726 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:400 IOs, 271624192 bytes 2023-12-10T19:08:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:500 IOs, 314051584 bytes 2023-12-10T19:08:35.030 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:600 IOs, 318478336 bytes 2023-12-10T19:08:35.397 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:700 IOs, 349281280 bytes 2023-12-10T19:08:35.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:800 IOs, 434507776 bytes 2023-12-10T19:08:35.892 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:900 IOs, 495106048 bytes 2023-12-10T19:08:36.101 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1000 IOs, 544792576 bytes 2023-12-10T19:08:36.262 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1100 IOs, 566687744 bytes 2023-12-10T19:08:36.416 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1200 IOs, 586443776 bytes 2023-12-10T19:08:36.530 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1300 IOs, 587051008 bytes 2023-12-10T19:08:36.751 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1400 IOs, 639111168 bytes 2023-12-10T19:08:36.928 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1500 IOs, 671938560 bytes 2023-12-10T19:08:37.129 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1600 IOs, 716565504 bytes 2023-12-10T19:08:37.331 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1700 IOs, 762702848 bytes 2023-12-10T19:08:37.448 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1800 IOs, 770553856 bytes 2023-12-10T19:08:37.714 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1900 IOs, 849623040 bytes 2023-12-10T19:08:37.913 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2000 IOs, 891244544 bytes 2023-12-10T19:08:38.078 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2100 IOs, 920083456 bytes 2023-12-10T19:08:38.227 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2200 IOs, 942475264 bytes 2023-12-10T19:08:38.438 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2300 IOs, 989618176 bytes 2023-12-10T19:08:38.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2400 IOs, 992961536 bytes 2023-12-10T19:08:38.797 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2500 IOs, 1056263168 bytes 2023-12-10T19:08:38.913 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2600 IOs, 1056902144 bytes 2023-12-10T19:08:39.095 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2700 IOs, 1097008128 bytes 2023-12-10T19:08:39.331 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2800 IOs, 1154430976 bytes 2023-12-10T19:08:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2900 IOs, 1239568384 bytes 2023-12-10T19:08:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3000 IOs, 1313393664 bytes 2023-12-10T19:08:40.105 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3100 IOs, 1365557248 bytes 2023-12-10T19:08:40.337 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3200 IOs, 1426638848 bytes 2023-12-10T19:08:40.461 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3300 IOs, 1434466304 bytes 2023-12-10T19:08:40.604 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3400 IOs, 1458321408 bytes 2023-12-10T19:08:40.761 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3500 IOs, 1480706048 bytes 2023-12-10T19:08:40.902 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3600 IOs, 1522440192 bytes 2023-12-10T19:08:41.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3700 IOs, 1558763520 bytes 2023-12-10T19:08:41.299 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3800 IOs, 1588410368 bytes 2023-12-10T19:08:41.513 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3900 IOs, 1633586176 bytes 2023-12-10T19:08:41.626 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4000 IOs, 1634384896 bytes 2023-12-10T19:08:41.846 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4100 IOs, 1692385280 bytes 2023-12-10T19:08:42.050 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4200 IOs, 1738987520 bytes 2023-12-10T19:08:42.232 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4300 IOs, 1773965312 bytes 2023-12-10T19:08:42.418 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4400 IOs, 1809477632 bytes 2023-12-10T19:08:42.587 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4500 IOs, 1860560896 bytes 2023-12-10T19:08:42.830 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4600 IOs, 1909585920 bytes 2023-12-10T19:08:43.050 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4700 IOs, 1966587904 bytes 2023-12-10T19:08:43.219 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4800 IOs, 1996197888 bytes 2023-12-10T19:08:43.324 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4900 IOs, 2006297600 bytes 2023-12-10T19:08:43.534 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5000 IOs, 2042328064 bytes 2023-12-10T19:08:43.694 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5100 IOs, 2067749888 bytes 2023-12-10T19:08:43.946 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5200 IOs, 2138015744 bytes 2023-12-10T19:08:44.194 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5300 IOs, 2212011008 bytes 2023-12-10T19:08:44.322 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5400 IOs, 2224453632 bytes 2023-12-10T19:08:44.515 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5500 IOs, 2276823040 bytes 2023-12-10T19:08:44.734 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5600 IOs, 2323395584 bytes 2023-12-10T19:08:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5700 IOs, 2344648704 bytes 2023-12-10T19:08:45.093 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5800 IOs, 2388445184 bytes 2023-12-10T19:08:45.226 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5900 IOs, 2397534208 bytes 2023-12-10T19:08:45.446 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6000 IOs, 2458694656 bytes 2023-12-10T19:08:45.633 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6100 IOs, 2497560576 bytes 2023-12-10T19:08:45.709 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=124 2023-12-10T19:08:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T19:08:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 124 -eq 124 ] 2023-12-10T19:08:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:08:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_35 2023-12-10T19:08:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image_and_enable_mirror cluster2 mirror image_35 snapshot 128M 2023-12-10T19:08:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:08:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:08:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:08:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:08:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_35 2023-12-10T19:08:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:08:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:08:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n snapshot ] 2023-12-10T19:08:45.711 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:08:45.711 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image cluster2 mirror image_35 128M 2023-12-10T19:08:45.711 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:08:45.711 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:08:45.711 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:08:45.711 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:08:45.711 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_35 2023-12-10T19:08:45.711 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:08:45.711 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local size=128 2023-12-10T19:08:45.711 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n 128M ] 2023-12-10T19:08:45.711 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ size=128M 2023-12-10T19:08:45.711 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:08:45.711 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_35 2023-12-10T19:08:45.757 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ is_pool_mirror_mode_image mirror 2023-12-10T19:08:45.757 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:08:45.757 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ image = image ] 2023-12-10T19:08:45.757 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:08:45.758 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ enable_mirror cluster2 mirror image_35 snapshot 2023-12-10T19:08:45.758 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:08:45.758 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:08:45.758 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_35 2023-12-10T19:08:45.758 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:08:45.758 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_35 snapshot 2023-12-10T19:08:46.141 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirroring enabled 2023-12-10T19:08:46.154 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 info mirror/image_35 2023-12-10T19:08:46.200 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_35': 2023-12-10T19:08:46.201 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:08:46.201 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:08:46.201 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 1 2023-12-10T19:08:46.201 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 141db396a5c8 2023-12-10T19:08:46.201 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.141db396a5c8 2023-12-10T19:08:46.201 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:08:46.201 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering 2023-12-10T19:08:46.201 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:08:46.201 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:08:46.201 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:08:45 2023 2023-12-10T19:08:46.201 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:08:45 2023 2023-12-10T19:08:46.201 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:08:45 2023 2023-12-10T19:08:46.201 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:08:46.201 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:08:46.201 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: fc4fbc1c-ab50-48d4-a59d-0c60a31269a4 2023-12-10T19:08:46.202 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: true 2023-12-10T19:08:46.204 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T19:08:46.204 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stress_write_image cluster2 mirror image_35 2023-12-10T19:08:46.204 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:08:46.205 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:08:46.205 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_35 2023-12-10T19:08:46.205 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-10T19:08:46.206 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local duration=6 2023-12-10T19:08:46.206 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T19:08:46.206 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_35 --debug-rbd=20 --debug-journaler=20 2023-12-10T19:08:46.384 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:100 IOs, 12530688 bytes 2023-12-10T19:08:46.637 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:200 IOs, 81926144 bytes 2023-12-10T19:08:46.911 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:300 IOs, 163501056 bytes 2023-12-10T19:08:47.250 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:400 IOs, 282676224 bytes 2023-12-10T19:08:47.467 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:500 IOs, 337778688 bytes 2023-12-10T19:08:47.603 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:600 IOs, 350834688 bytes 2023-12-10T19:08:47.862 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:700 IOs, 424411136 bytes 2023-12-10T19:08:48.114 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:800 IOs, 498641920 bytes 2023-12-10T19:08:48.318 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:900 IOs, 550153216 bytes 2023-12-10T19:08:48.500 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1000 IOs, 578357248 bytes 2023-12-10T19:08:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1100 IOs, 630671360 bytes 2023-12-10T19:08:48.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1200 IOs, 711737344 bytes 2023-12-10T19:08:49.136 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1300 IOs, 733029376 bytes 2023-12-10T19:08:49.299 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1400 IOs, 758913024 bytes 2023-12-10T19:08:49.481 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1500 IOs, 788270080 bytes 2023-12-10T19:08:49.615 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1600 IOs, 788889600 bytes 2023-12-10T19:08:49.770 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1700 IOs, 810654720 bytes 2023-12-10T19:08:49.974 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1800 IOs, 858793984 bytes 2023-12-10T19:08:50.134 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1900 IOs, 877092864 bytes 2023-12-10T19:08:50.358 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2000 IOs, 937872384 bytes 2023-12-10T19:08:50.504 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2100 IOs, 938477568 bytes 2023-12-10T19:08:50.710 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2200 IOs, 988082176 bytes 2023-12-10T19:08:50.986 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2300 IOs, 1071628288 bytes 2023-12-10T19:08:51.145 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2400 IOs, 1114100736 bytes 2023-12-10T19:08:51.348 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2500 IOs, 1143535616 bytes 2023-12-10T19:08:51.511 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2600 IOs, 1197426688 bytes 2023-12-10T19:08:51.778 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2700 IOs, 1274563584 bytes 2023-12-10T19:08:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2800 IOs, 1374397440 bytes 2023-12-10T19:08:52.212 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=124 2023-12-10T19:08:52.212 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T19:08:52.212 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 124 -eq 124 ] 2023-12-10T19:08:52.212 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:08:52.212 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_36 2023-12-10T19:08:52.212 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image_and_enable_mirror cluster2 mirror image_36 snapshot 128M 2023-12-10T19:08:52.213 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:08:52.213 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:08:52.213 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:08:52.213 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:08:52.213 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_36 2023-12-10T19:08:52.213 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:08:52.213 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:08:52.213 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n snapshot ] 2023-12-10T19:08:52.213 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:08:52.213 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image cluster2 mirror image_36 128M 2023-12-10T19:08:52.213 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:08:52.213 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:08:52.213 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:08:52.213 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:08:52.214 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_36 2023-12-10T19:08:52.214 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:08:52.214 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local size=128 2023-12-10T19:08:52.214 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n 128M ] 2023-12-10T19:08:52.214 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ size=128M 2023-12-10T19:08:52.214 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:08:52.214 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_36 2023-12-10T19:08:52.261 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ is_pool_mirror_mode_image mirror 2023-12-10T19:08:52.262 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:08:52.262 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ image = image ] 2023-12-10T19:08:52.262 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:08:52.262 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ enable_mirror cluster2 mirror image_36 snapshot 2023-12-10T19:08:52.262 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:08:52.262 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:08:52.262 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_36 2023-12-10T19:08:52.262 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:08:52.262 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_36 snapshot 2023-12-10T19:08:52.570 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirroring enabled 2023-12-10T19:08:52.581 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 info mirror/image_36 2023-12-10T19:08:52.622 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_36': 2023-12-10T19:08:52.622 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:08:52.622 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:08:52.622 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 1 2023-12-10T19:08:52.622 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 1427874d626e 2023-12-10T19:08:52.622 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.1427874d626e 2023-12-10T19:08:52.623 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:08:52.623 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering 2023-12-10T19:08:52.623 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:08:52.623 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:08:52.623 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:08:52 2023 2023-12-10T19:08:52.623 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:08:52 2023 2023-12-10T19:08:52.623 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:08:52 2023 2023-12-10T19:08:52.623 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:08:52.623 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:08:52.623 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: ad2cd5bc-da17-44b5-9afd-c05bcf6f8746 2023-12-10T19:08:52.623 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: true 2023-12-10T19:08:52.626 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T19:08:52.626 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stress_write_image cluster2 mirror image_36 2023-12-10T19:08:52.626 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:08:52.626 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:08:52.626 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_36 2023-12-10T19:08:52.626 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-10T19:08:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local duration=6 2023-12-10T19:08:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T19:08:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_36 --debug-rbd=20 --debug-journaler=20 2023-12-10T19:08:52.826 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:100 IOs, 29466624 bytes 2023-12-10T19:08:52.983 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:200 IOs, 53752832 bytes 2023-12-10T19:08:53.198 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:300 IOs, 125493248 bytes 2023-12-10T19:08:53.425 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:400 IOs, 190126080 bytes 2023-12-10T19:08:53.624 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:500 IOs, 223875072 bytes 2023-12-10T19:08:53.870 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:600 IOs, 291243008 bytes 2023-12-10T19:08:54.000 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:700 IOs, 300360704 bytes 2023-12-10T19:08:54.274 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:800 IOs, 386651136 bytes 2023-12-10T19:08:54.638 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:900 IOs, 505601024 bytes 2023-12-10T19:08:54.898 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1000 IOs, 580022272 bytes 2023-12-10T19:08:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1100 IOs, 587233280 bytes 2023-12-10T19:08:55.199 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1200 IOs, 615557120 bytes 2023-12-10T19:08:55.333 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1300 IOs, 626816000 bytes 2023-12-10T19:08:55.459 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1400 IOs, 631636992 bytes 2023-12-10T19:08:55.614 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1500 IOs, 657225728 bytes 2023-12-10T19:08:55.805 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1600 IOs, 701244416 bytes 2023-12-10T19:08:56.061 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1700 IOs, 766666752 bytes 2023-12-10T19:08:56.363 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1800 IOs, 859902976 bytes 2023-12-10T19:08:56.610 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1900 IOs, 927281152 bytes 2023-12-10T19:08:56.789 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2000 IOs, 960547840 bytes 2023-12-10T19:08:56.904 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2100 IOs, 962159616 bytes 2023-12-10T19:08:57.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2200 IOs, 1037907968 bytes 2023-12-10T19:08:57.392 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2300 IOs, 1100783616 bytes 2023-12-10T19:08:57.615 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2400 IOs, 1156433920 bytes 2023-12-10T19:08:57.826 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2500 IOs, 1202882560 bytes 2023-12-10T19:08:57.956 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2600 IOs, 1214246912 bytes 2023-12-10T19:08:58.177 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2700 IOs, 1264908288 bytes 2023-12-10T19:08:58.374 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2800 IOs, 1313447936 bytes 2023-12-10T19:08:58.533 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2900 IOs, 1337310208 bytes 2023-12-10T19:08:58.633 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=124 2023-12-10T19:08:58.633 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T19:08:58.633 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 124 -eq 124 ] 2023-12-10T19:08:58.633 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:08:58.634 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_37 2023-12-10T19:08:58.634 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image_and_enable_mirror cluster2 mirror image_37 snapshot 128M 2023-12-10T19:08:58.634 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:08:58.634 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:08:58.634 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:08:58.634 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:08:58.634 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_37 2023-12-10T19:08:58.634 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:08:58.634 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:08:58.634 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n snapshot ] 2023-12-10T19:08:58.634 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:08:58.634 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image cluster2 mirror image_37 128M 2023-12-10T19:08:58.634 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:08:58.634 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:08:58.635 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:08:58.635 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:08:58.635 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_37 2023-12-10T19:08:58.635 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:08:58.635 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local size=128 2023-12-10T19:08:58.635 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n 128M ] 2023-12-10T19:08:58.635 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ size=128M 2023-12-10T19:08:58.635 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:08:58.635 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_37 2023-12-10T19:08:58.679 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ is_pool_mirror_mode_image mirror 2023-12-10T19:08:58.679 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:08:58.679 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ image = image ] 2023-12-10T19:08:58.679 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:08:58.679 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ enable_mirror cluster2 mirror image_37 snapshot 2023-12-10T19:08:58.679 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:08:58.679 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:08:58.679 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_37 2023-12-10T19:08:58.679 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:08:58.679 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_37 snapshot 2023-12-10T19:08:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirroring enabled 2023-12-10T19:08:59.600 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 info mirror/image_37 2023-12-10T19:08:59.645 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_37': 2023-12-10T19:08:59.645 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:08:59.646 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:08:59.646 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 1 2023-12-10T19:08:59.646 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 142ff86626be 2023-12-10T19:08:59.646 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.142ff86626be 2023-12-10T19:08:59.646 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:08:59.646 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering 2023-12-10T19:08:59.646 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:08:59.646 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:08:59.646 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:08:58 2023 2023-12-10T19:08:59.646 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:08:58 2023 2023-12-10T19:08:59.646 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:08:58 2023 2023-12-10T19:08:59.646 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:08:59.646 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:08:59.647 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: a4d8c861-aa4e-4c85-9970-1f362ca1c6b6 2023-12-10T19:08:59.647 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: true 2023-12-10T19:08:59.649 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T19:08:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stress_write_image cluster2 mirror image_37 2023-12-10T19:08:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:08:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:08:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_37 2023-12-10T19:08:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-10T19:08:59.651 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local duration=14 2023-12-10T19:08:59.651 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T19:08:59.651 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_37 --debug-rbd=20 --debug-journaler=20 2023-12-10T19:08:59.852 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:100 IOs, 29214720 bytes 2023-12-10T19:08:59.967 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:200 IOs, 33855488 bytes 2023-12-10T19:09:00.122 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:300 IOs, 72156160 bytes 2023-12-10T19:09:00.486 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:400 IOs, 198872064 bytes 2023-12-10T19:09:00.664 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:500 IOs, 237579264 bytes 2023-12-10T19:09:00.849 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:600 IOs, 257421312 bytes 2023-12-10T19:09:01.023 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:700 IOs, 287709184 bytes 2023-12-10T19:09:01.211 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:800 IOs, 331504640 bytes 2023-12-10T19:09:01.407 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:900 IOs, 375804928 bytes 2023-12-10T19:09:01.579 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1000 IOs, 406766592 bytes 2023-12-10T19:09:01.741 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1100 IOs, 426867712 bytes 2023-12-10T19:09:02.000 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1200 IOs, 497384448 bytes 2023-12-10T19:09:02.211 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1300 IOs, 554705920 bytes 2023-12-10T19:09:02.365 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1400 IOs, 580638720 bytes 2023-12-10T19:09:02.482 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1500 IOs, 581842944 bytes 2023-12-10T19:09:02.621 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1600 IOs, 596813824 bytes 2023-12-10T19:09:02.796 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1700 IOs, 647607296 bytes 2023-12-10T19:09:03.183 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1800 IOs, 767416320 bytes 2023-12-10T19:09:03.340 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1900 IOs, 798954496 bytes 2023-12-10T19:09:03.560 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2000 IOs, 852514816 bytes 2023-12-10T19:09:03.760 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2100 IOs, 907826176 bytes 2023-12-10T19:09:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2200 IOs, 994174976 bytes 2023-12-10T19:09:04.271 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2300 IOs, 1043198976 bytes 2023-12-10T19:09:04.507 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2400 IOs, 1102649344 bytes 2023-12-10T19:09:04.723 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2500 IOs, 1154857984 bytes 2023-12-10T19:09:04.847 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6773057256964802) > weight (0.25) 2023-12-10T19:09:04.847 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.5376886833250993) > weight (0.25) 2023-12-10T19:09:04.847 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2023-12-10T19:09:04.847 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2023-12-10T19:09:04.847 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2023-12-10T19:09:04.850 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi076.stderr:2023-12-10T19:09:04.856+0000 7f6ecbe37640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 49684) UID: 1000 2023-12-10T19:09:04.919 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2600 IOs, 1191158784 bytes 2023-12-10T19:09:05.072 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2700 IOs, 1225667584 bytes 2023-12-10T19:09:05.224 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2800 IOs, 1232708608 bytes 2023-12-10T19:09:05.404 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2900 IOs, 1268255744 bytes 2023-12-10T19:09:05.526 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3000 IOs, 1291336704 bytes 2023-12-10T19:09:05.789 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3100 IOs, 1355228160 bytes 2023-12-10T19:09:06.020 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3200 IOs, 1416545280 bytes 2023-12-10T19:09:06.190 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3300 IOs, 1449069568 bytes 2023-12-10T19:09:06.555 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3400 IOs, 1582617600 bytes 2023-12-10T19:09:06.758 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3500 IOs, 1634033664 bytes 2023-12-10T19:09:06.901 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3600 IOs, 1647049728 bytes 2023-12-10T19:09:07.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3700 IOs, 1667982336 bytes 2023-12-10T19:09:07.296 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3800 IOs, 1746119680 bytes 2023-12-10T19:09:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3900 IOs, 1796632576 bytes 2023-12-10T19:09:07.762 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4000 IOs, 1869181952 bytes 2023-12-10T19:09:07.929 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4100 IOs, 1907254272 bytes 2023-12-10T19:09:08.194 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4200 IOs, 1978660864 bytes 2023-12-10T19:09:08.382 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4300 IOs, 2024855552 bytes 2023-12-10T19:09:08.568 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4400 IOs, 2077980672 bytes 2023-12-10T19:09:08.756 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4500 IOs, 2098927616 bytes 2023-12-10T19:09:08.940 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4600 IOs, 2137250816 bytes 2023-12-10T19:09:09.108 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4700 IOs, 2160850944 bytes 2023-12-10T19:09:09.292 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4800 IOs, 2197199872 bytes 2023-12-10T19:09:09.476 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4900 IOs, 2239617024 bytes 2023-12-10T19:09:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5000 IOs, 2261478400 bytes 2023-12-10T19:09:09.764 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5100 IOs, 2277462016 bytes 2023-12-10T19:09:09.935 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5200 IOs, 2308956160 bytes 2023-12-10T19:09:10.066 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5300 IOs, 2339290112 bytes 2023-12-10T19:09:10.274 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5400 IOs, 2378339328 bytes 2023-12-10T19:09:10.468 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5500 IOs, 2426721280 bytes 2023-12-10T19:09:10.656 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5600 IOs, 2468016128 bytes 2023-12-10T19:09:10.883 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5700 IOs, 2531991552 bytes 2023-12-10T19:09:11.058 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5800 IOs, 2554736640 bytes 2023-12-10T19:09:11.263 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5900 IOs, 2632196096 bytes 2023-12-10T19:09:11.484 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6000 IOs, 2664263680 bytes 2023-12-10T19:09:11.762 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6100 IOs, 2774465536 bytes 2023-12-10T19:09:11.848 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2023-12-10T19:09:11.848 INFO:teuthology.orchestra.run:waiting for 600 2023-12-10T19:09:11.848 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2023-12-10T19:09:11.848 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2023-12-10T19:09:11.849 DEBUG:teuthology.orchestra.run.smithi076:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2023-12-10T19:09:11.851 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2023-12-10T19:09:11.851 INFO:tasks.rbd_mirror_thrash:waiting for 98 secs before thrashing 2023-12-10T19:09:11.950 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6200 IOs, 2787315712 bytes 2023-12-10T19:09:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6300 IOs, 2810058752 bytes 2023-12-10T19:09:12.342 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6400 IOs, 2857778176 bytes 2023-12-10T19:09:12.604 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6500 IOs, 2924558336 bytes 2023-12-10T19:09:12.731 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6600 IOs, 2933076992 bytes 2023-12-10T19:09:12.836 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6700 IOs, 2933682176 bytes 2023-12-10T19:09:13.041 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6800 IOs, 2982995968 bytes 2023-12-10T19:09:13.209 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6900 IOs, 3014483968 bytes 2023-12-10T19:09:13.345 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:7000 IOs, 3028936704 bytes 2023-12-10T19:09:13.570 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:7100 IOs, 3086635008 bytes 2023-12-10T19:09:13.655 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=124 2023-12-10T19:09:13.656 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T19:09:13.656 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 124 -eq 124 ] 2023-12-10T19:09:13.656 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:09:13.656 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_38 2023-12-10T19:09:13.656 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image_and_enable_mirror cluster2 mirror image_38 snapshot 128M 2023-12-10T19:09:13.656 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:09:13.656 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:09:13.656 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:09:13.656 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:09:13.656 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_38 2023-12-10T19:09:13.656 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:09:13.656 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:09:13.656 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n snapshot ] 2023-12-10T19:09:13.656 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:09:13.656 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image cluster2 mirror image_38 128M 2023-12-10T19:09:13.657 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:09:13.657 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:09:13.657 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:09:13.657 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:09:13.657 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_38 2023-12-10T19:09:13.657 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:09:13.657 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local size=128 2023-12-10T19:09:13.657 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n 128M ] 2023-12-10T19:09:13.657 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ size=128M 2023-12-10T19:09:13.657 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:09:13.657 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_38 2023-12-10T19:09:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ is_pool_mirror_mode_image mirror 2023-12-10T19:09:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:09:13.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ image = image ] 2023-12-10T19:09:13.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:09:13.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ enable_mirror cluster2 mirror image_38 snapshot 2023-12-10T19:09:13.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:09:13.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:09:13.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_38 2023-12-10T19:09:13.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:09:13.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_38 snapshot 2023-12-10T19:09:14.665 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirroring enabled 2023-12-10T19:09:14.676 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 info mirror/image_38 2023-12-10T19:09:14.722 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_38': 2023-12-10T19:09:14.722 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:09:14.723 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:09:14.723 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 1 2023-12-10T19:09:14.723 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 1443fa195f82 2023-12-10T19:09:14.723 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.1443fa195f82 2023-12-10T19:09:14.723 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:09:14.723 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering 2023-12-10T19:09:14.723 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:09:14.723 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:09:14.723 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:09:13 2023 2023-12-10T19:09:14.723 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:09:13 2023 2023-12-10T19:09:14.723 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:09:13 2023 2023-12-10T19:09:14.723 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:09:14.723 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:09:14.723 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: 82e2c879-d2a6-4227-b47b-c52a54933f28 2023-12-10T19:09:14.723 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: true 2023-12-10T19:09:14.726 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T19:09:14.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stress_write_image cluster2 mirror image_38 2023-12-10T19:09:14.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:09:14.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:09:14.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_38 2023-12-10T19:09:14.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-10T19:09:14.728 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local duration=11 2023-12-10T19:09:14.728 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T19:09:14.728 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_38 --debug-rbd=20 --debug-journaler=20 2023-12-10T19:09:14.877 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:100 IOs, 590848 bytes 2023-12-10T19:09:15.118 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:200 IOs, 69214208 bytes 2023-12-10T19:09:15.304 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:300 IOs, 129477632 bytes 2023-12-10T19:09:15.647 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:400 IOs, 223712256 bytes 2023-12-10T19:09:15.849 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:500 IOs, 261033984 bytes 2023-12-10T19:09:15.994 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:600 IOs, 279122944 bytes 2023-12-10T19:09:16.108 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:700 IOs, 279733248 bytes 2023-12-10T19:09:16.230 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:800 IOs, 287820800 bytes 2023-12-10T19:09:16.411 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:900 IOs, 328561664 bytes 2023-12-10T19:09:16.556 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1000 IOs, 340403200 bytes 2023-12-10T19:09:16.814 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1100 IOs, 415587328 bytes 2023-12-10T19:09:16.999 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1200 IOs, 449909760 bytes 2023-12-10T19:09:17.188 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1300 IOs, 490683392 bytes 2023-12-10T19:09:17.384 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1400 IOs, 539251712 bytes 2023-12-10T19:09:17.511 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1500 IOs, 542123008 bytes 2023-12-10T19:09:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1600 IOs, 546822144 bytes 2023-12-10T19:09:17.841 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1700 IOs, 592740352 bytes 2023-12-10T19:09:17.970 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1800 IOs, 600642560 bytes 2023-12-10T19:09:18.197 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1900 IOs, 685713408 bytes 2023-12-10T19:09:18.392 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2000 IOs, 709555200 bytes 2023-12-10T19:09:18.536 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2100 IOs, 727227392 bytes 2023-12-10T19:09:18.670 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2200 IOs, 745459712 bytes 2023-12-10T19:09:18.792 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2300 IOs, 757009408 bytes 2023-12-10T19:09:18.952 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2400 IOs, 782301184 bytes 2023-12-10T19:09:19.209 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2500 IOs, 862358528 bytes 2023-12-10T19:09:19.445 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2600 IOs, 932143104 bytes 2023-12-10T19:09:19.574 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2700 IOs, 932767744 bytes 2023-12-10T19:09:19.760 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2800 IOs, 971774976 bytes 2023-12-10T19:09:19.961 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2900 IOs, 1019457536 bytes 2023-12-10T19:09:20.089 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3000 IOs, 1028646912 bytes 2023-12-10T19:09:20.330 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3100 IOs, 1087156224 bytes 2023-12-10T19:09:20.508 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3200 IOs, 1125853184 bytes 2023-12-10T19:09:20.707 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3300 IOs, 1171993600 bytes 2023-12-10T19:09:20.939 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3400 IOs, 1231674368 bytes 2023-12-10T19:09:21.072 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3500 IOs, 1239561216 bytes 2023-12-10T19:09:21.262 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3600 IOs, 1277740032 bytes 2023-12-10T19:09:21.406 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3700 IOs, 1299947520 bytes 2023-12-10T19:09:21.658 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3800 IOs, 1371334656 bytes 2023-12-10T19:09:21.841 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3900 IOs, 1399793664 bytes 2023-12-10T19:09:22.162 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4000 IOs, 1513359360 bytes 2023-12-10T19:09:22.324 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4100 IOs, 1537730560 bytes 2023-12-10T19:09:22.494 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4200 IOs, 1572188160 bytes 2023-12-10T19:09:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4300 IOs, 1572808704 bytes 2023-12-10T19:09:22.796 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4400 IOs, 1610030080 bytes 2023-12-10T19:09:22.942 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4500 IOs, 1631230976 bytes 2023-12-10T19:09:23.135 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4600 IOs, 1673693184 bytes 2023-12-10T19:09:23.291 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4700 IOs, 1697526784 bytes 2023-12-10T19:09:23.460 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4800 IOs, 1723011072 bytes 2023-12-10T19:09:23.613 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4900 IOs, 1744984064 bytes 2023-12-10T19:09:23.778 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5000 IOs, 1766656000 bytes 2023-12-10T19:09:23.896 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5100 IOs, 1767530496 bytes 2023-12-10T19:09:24.181 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5200 IOs, 1847895040 bytes 2023-12-10T19:09:24.283 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5300 IOs, 1849950208 bytes 2023-12-10T19:09:24.466 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5400 IOs, 1884840960 bytes 2023-12-10T19:09:24.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5500 IOs, 1932115968 bytes 2023-12-10T19:09:24.786 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5600 IOs, 1932711936 bytes 2023-12-10T19:09:24.966 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5700 IOs, 1967893504 bytes 2023-12-10T19:09:25.116 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5800 IOs, 1998073856 bytes 2023-12-10T19:09:25.375 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5900 IOs, 2068912128 bytes 2023-12-10T19:09:25.562 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6000 IOs, 2093893632 bytes 2023-12-10T19:09:25.733 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=124 2023-12-10T19:09:25.734 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T19:09:25.734 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 124 -eq 124 ] 2023-12-10T19:09:25.734 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:09:25.734 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_39 2023-12-10T19:09:25.734 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image_and_enable_mirror cluster2 mirror image_39 snapshot 128M 2023-12-10T19:09:25.734 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:09:25.734 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:09:25.734 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:09:25.734 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:09:25.734 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_39 2023-12-10T19:09:25.734 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:09:25.734 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:09:25.735 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n snapshot ] 2023-12-10T19:09:25.735 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:09:25.735 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image cluster2 mirror image_39 128M 2023-12-10T19:09:25.735 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:09:25.735 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:09:25.735 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:09:25.735 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:09:25.735 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_39 2023-12-10T19:09:25.735 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:09:25.735 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local size=128 2023-12-10T19:09:25.735 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n 128M ] 2023-12-10T19:09:25.735 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ size=128M 2023-12-10T19:09:25.735 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:09:25.735 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_39 2023-12-10T19:09:25.781 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ is_pool_mirror_mode_image mirror 2023-12-10T19:09:25.782 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:09:25.782 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ image = image ] 2023-12-10T19:09:25.782 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:09:25.782 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ enable_mirror cluster2 mirror image_39 snapshot 2023-12-10T19:09:25.782 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:09:25.782 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:09:25.782 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_39 2023-12-10T19:09:25.782 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:09:25.782 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_39 snapshot 2023-12-10T19:09:26.166 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirroring enabled 2023-12-10T19:09:26.176 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 info mirror/image_39 2023-12-10T19:09:26.223 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_39': 2023-12-10T19:09:26.223 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:09:26.223 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:09:26.223 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 1 2023-12-10T19:09:26.223 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 144c2b339030 2023-12-10T19:09:26.223 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.144c2b339030 2023-12-10T19:09:26.223 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:09:26.223 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering 2023-12-10T19:09:26.223 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:09:26.223 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:09:26.223 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:09:25 2023 2023-12-10T19:09:26.224 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:09:25 2023 2023-12-10T19:09:26.224 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:09:25 2023 2023-12-10T19:09:26.224 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:09:26.224 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:09:26.224 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: b649fd8b-2f2a-4f7d-94c5-3e9e6a6f4c99 2023-12-10T19:09:26.224 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: true 2023-12-10T19:09:26.227 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T19:09:26.227 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stress_write_image cluster2 mirror image_39 2023-12-10T19:09:26.227 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:09:26.227 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:09:26.227 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_39 2023-12-10T19:09:26.227 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-10T19:09:26.228 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local duration=11 2023-12-10T19:09:26.228 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T19:09:26.228 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_39 --debug-rbd=20 --debug-journaler=20 2023-12-10T19:09:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:100 IOs, 8010752 bytes 2023-12-10T19:09:26.540 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:200 IOs, 34655232 bytes 2023-12-10T19:09:26.737 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:300 IOs, 102330368 bytes 2023-12-10T19:09:26.905 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:400 IOs, 126646272 bytes 2023-12-10T19:09:27.173 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:500 IOs, 200050688 bytes 2023-12-10T19:09:27.311 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:600 IOs, 214902784 bytes 2023-12-10T19:09:27.485 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:700 IOs, 267909120 bytes 2023-12-10T19:09:27.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:800 IOs, 302425088 bytes 2023-12-10T19:09:27.834 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:900 IOs, 313506816 bytes 2023-12-10T19:09:28.075 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1000 IOs, 381475840 bytes 2023-12-10T19:09:28.242 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1100 IOs, 408536064 bytes 2023-12-10T19:09:28.428 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1200 IOs, 465916928 bytes 2023-12-10T19:09:28.610 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1300 IOs, 467437568 bytes 2023-12-10T19:09:28.746 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1400 IOs, 472709120 bytes 2023-12-10T19:09:28.906 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1500 IOs, 495833088 bytes 2023-12-10T19:09:29.094 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1600 IOs, 537031680 bytes 2023-12-10T19:09:29.305 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1700 IOs, 586241024 bytes 2023-12-10T19:09:29.437 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1800 IOs, 598089728 bytes 2023-12-10T19:09:29.621 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1900 IOs, 640239616 bytes 2023-12-10T19:09:29.877 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2000 IOs, 714883072 bytes 2023-12-10T19:09:30.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2100 IOs, 755515392 bytes 2023-12-10T19:09:30.308 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2200 IOs, 814092288 bytes 2023-12-10T19:09:30.546 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2300 IOs, 887353344 bytes 2023-12-10T19:09:30.798 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2400 IOs, 955185152 bytes 2023-12-10T19:09:30.992 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2500 IOs, 1001200640 bytes 2023-12-10T19:09:31.179 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2600 IOs, 1034640384 bytes 2023-12-10T19:09:31.393 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2700 IOs, 1088588800 bytes 2023-12-10T19:09:31.606 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2800 IOs, 1137514496 bytes 2023-12-10T19:09:31.784 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2900 IOs, 1183107072 bytes 2023-12-10T19:09:31.954 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3000 IOs, 1209424896 bytes 2023-12-10T19:09:32.099 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3100 IOs, 1217554432 bytes 2023-12-10T19:09:32.291 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3200 IOs, 1256233984 bytes 2023-12-10T19:09:32.412 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3300 IOs, 1265778688 bytes 2023-12-10T19:09:32.618 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3400 IOs, 1312879616 bytes 2023-12-10T19:09:32.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3500 IOs, 1359823872 bytes 2023-12-10T19:09:33.000 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3600 IOs, 1401143296 bytes 2023-12-10T19:09:33.171 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3700 IOs, 1428881408 bytes 2023-12-10T19:09:33.374 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3800 IOs, 1465367552 bytes 2023-12-10T19:09:33.505 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3900 IOs, 1481236480 bytes 2023-12-10T19:09:33.743 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4000 IOs, 1523881984 bytes 2023-12-10T19:09:33.991 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4100 IOs, 1592437760 bytes 2023-12-10T19:09:34.138 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4200 IOs, 1610950656 bytes 2023-12-10T19:09:34.304 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4300 IOs, 1633678336 bytes 2023-12-10T19:09:34.470 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4400 IOs, 1655465984 bytes 2023-12-10T19:09:34.685 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4500 IOs, 1718028288 bytes 2023-12-10T19:09:34.945 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4600 IOs, 1767587840 bytes 2023-12-10T19:09:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4700 IOs, 1840535552 bytes 2023-12-10T19:09:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4800 IOs, 1861146624 bytes 2023-12-10T19:09:35.576 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4900 IOs, 1897696256 bytes 2023-12-10T19:09:35.777 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5000 IOs, 1940209664 bytes 2023-12-10T19:09:35.963 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5100 IOs, 1973645312 bytes 2023-12-10T19:09:36.206 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5200 IOs, 2025174016 bytes 2023-12-10T19:09:36.355 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5300 IOs, 2040941568 bytes 2023-12-10T19:09:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5400 IOs, 2117879808 bytes 2023-12-10T19:09:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5500 IOs, 2145147904 bytes 2023-12-10T19:09:36.853 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5370572929680039) > weight (0.25) 2023-12-10T19:09:36.853 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.8785966597248909) > weight (0.25) 2023-12-10T19:09:36.853 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.3017410328362494) > weight (0.25) 2023-12-10T19:09:36.853 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.7750420608435316) > weight (0.25) 2023-12-10T19:09:36.853 INFO:tasks.rbd_mirror_thrash:waiting for 75 secs before thrashing 2023-12-10T19:09:36.972 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5600 IOs, 2181835776 bytes 2023-12-10T19:09:37.143 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5700 IOs, 2207489024 bytes 2023-12-10T19:09:37.235 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=124 2023-12-10T19:09:37.236 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T19:09:37.236 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 124 -eq 124 ] 2023-12-10T19:09:37.236 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:09:37.236 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_40 2023-12-10T19:09:37.236 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image_and_enable_mirror cluster2 mirror image_40 snapshot 128M 2023-12-10T19:09:37.236 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:09:37.236 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:09:37.236 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:09:37.236 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:09:37.236 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_40 2023-12-10T19:09:37.236 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:09:37.236 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:09:37.236 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n snapshot ] 2023-12-10T19:09:37.236 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:09:37.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image cluster2 mirror image_40 128M 2023-12-10T19:09:37.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:09:37.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:09:37.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:09:37.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:09:37.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_40 2023-12-10T19:09:37.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:09:37.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local size=128 2023-12-10T19:09:37.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n 128M ] 2023-12-10T19:09:37.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ size=128M 2023-12-10T19:09:37.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:09:37.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_40 2023-12-10T19:09:37.298 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ is_pool_mirror_mode_image mirror 2023-12-10T19:09:37.298 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:09:37.298 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ image = image ] 2023-12-10T19:09:37.298 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:09:37.298 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ enable_mirror cluster2 mirror image_40 snapshot 2023-12-10T19:09:37.298 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:09:37.298 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:09:37.298 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_40 2023-12-10T19:09:37.298 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:09:37.298 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_40 snapshot 2023-12-10T19:09:37.601 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirroring enabled 2023-12-10T19:09:37.614 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 info mirror/image_40 2023-12-10T19:09:37.662 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_40': 2023-12-10T19:09:37.662 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:09:37.662 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:09:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 1 2023-12-10T19:09:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 1454baf79504 2023-12-10T19:09:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.1454baf79504 2023-12-10T19:09:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:09:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering 2023-12-10T19:09:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:09:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:09:37.664 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:09:37 2023 2023-12-10T19:09:37.664 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:09:37 2023 2023-12-10T19:09:37.664 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:09:37 2023 2023-12-10T19:09:37.664 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:09:37.664 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:09:37.664 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: 36d7ae4a-9a61-4f12-8a89-e0b759776b33 2023-12-10T19:09:37.664 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: true 2023-12-10T19:09:37.666 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T19:09:37.667 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stress_write_image cluster2 mirror image_40 2023-12-10T19:09:37.667 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:09:37.667 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:09:37.667 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_40 2023-12-10T19:09:37.667 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-10T19:09:37.668 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local duration=11 2023-12-10T19:09:37.668 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T19:09:37.668 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_40 --debug-rbd=20 --debug-journaler=20 2023-12-10T19:09:38.170 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:100 IOs, 63581184 bytes 2023-12-10T19:09:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:200 IOs, 84354048 bytes 2023-12-10T19:09:38.518 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:300 IOs, 121078784 bytes 2023-12-10T19:09:38.749 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:400 IOs, 180040704 bytes 2023-12-10T19:09:38.866 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:500 IOs, 183340032 bytes 2023-12-10T19:09:39.145 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:600 IOs, 269293568 bytes 2023-12-10T19:09:39.380 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:700 IOs, 337128448 bytes 2023-12-10T19:09:39.626 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:800 IOs, 388037632 bytes 2023-12-10T19:09:39.822 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:900 IOs, 442288128 bytes 2023-12-10T19:09:40.006 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1000 IOs, 475057152 bytes 2023-12-10T19:09:40.268 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1100 IOs, 549958656 bytes 2023-12-10T19:09:40.386 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1200 IOs, 559591424 bytes 2023-12-10T19:09:40.554 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1300 IOs, 585889792 bytes 2023-12-10T19:09:40.784 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1400 IOs, 662403072 bytes 2023-12-10T19:09:40.939 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1500 IOs, 662995968 bytes 2023-12-10T19:09:41.177 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1600 IOs, 728888320 bytes 2023-12-10T19:09:41.400 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1700 IOs, 783208448 bytes 2023-12-10T19:09:41.588 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1800 IOs, 822647808 bytes 2023-12-10T19:09:41.765 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1900 IOs, 858949632 bytes 2023-12-10T19:09:41.934 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2000 IOs, 882312192 bytes 2023-12-10T19:09:42.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2100 IOs, 920350720 bytes 2023-12-10T19:09:42.337 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2200 IOs, 975961088 bytes 2023-12-10T19:09:42.480 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2300 IOs, 989260800 bytes 2023-12-10T19:09:42.611 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2400 IOs, 1007493120 bytes 2023-12-10T19:09:42.758 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2500 IOs, 1017017344 bytes 2023-12-10T19:09:42.866 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2600 IOs, 1018939392 bytes 2023-12-10T19:09:42.988 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2700 IOs, 1028169728 bytes 2023-12-10T19:09:43.209 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2800 IOs, 1077292032 bytes 2023-12-10T19:09:43.449 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2900 IOs, 1140691968 bytes 2023-12-10T19:09:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3000 IOs, 1161277440 bytes 2023-12-10T19:09:43.860 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3100 IOs, 1235611648 bytes 2023-12-10T19:09:44.165 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3200 IOs, 1330473984 bytes 2023-12-10T19:09:44.476 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3300 IOs, 1393199104 bytes 2023-12-10T19:09:44.631 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3400 IOs, 1420925952 bytes 2023-12-10T19:09:44.854 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3500 IOs, 1470721024 bytes 2023-12-10T19:09:45.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3600 IOs, 1503822848 bytes 2023-12-10T19:09:45.232 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3700 IOs, 1552955392 bytes 2023-12-10T19:09:45.426 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3800 IOs, 1602873344 bytes 2023-12-10T19:09:45.587 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3900 IOs, 1611338752 bytes 2023-12-10T19:09:45.772 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4000 IOs, 1646459904 bytes 2023-12-10T19:09:45.945 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4100 IOs, 1679970304 bytes 2023-12-10T19:09:46.162 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4200 IOs, 1734497280 bytes 2023-12-10T19:09:46.402 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4300 IOs, 1799992320 bytes 2023-12-10T19:09:46.559 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4400 IOs, 1825677312 bytes 2023-12-10T19:09:46.808 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4500 IOs, 1894246400 bytes 2023-12-10T19:09:47.054 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4600 IOs, 1959555072 bytes 2023-12-10T19:09:47.334 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4700 IOs, 2042586112 bytes 2023-12-10T19:09:47.505 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4800 IOs, 2076693504 bytes 2023-12-10T19:09:47.669 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4900 IOs, 2082382848 bytes 2023-12-10T19:09:47.883 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5000 IOs, 2125238272 bytes 2023-12-10T19:09:48.040 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5100 IOs, 2139065344 bytes 2023-12-10T19:09:48.222 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5200 IOs, 2173810688 bytes 2023-12-10T19:09:48.492 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5300 IOs, 2262233088 bytes 2023-12-10T19:09:48.672 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=124 2023-12-10T19:09:48.672 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T19:09:48.672 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 124 -eq 124 ] 2023-12-10T19:09:48.672 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:09:48.672 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_41 2023-12-10T19:09:48.672 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image_and_enable_mirror cluster2 mirror image_41 snapshot 128M 2023-12-10T19:09:48.673 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:09:48.673 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:09:48.673 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:09:48.673 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:09:48.673 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_41 2023-12-10T19:09:48.673 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:09:48.673 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:09:48.673 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n snapshot ] 2023-12-10T19:09:48.673 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:09:48.673 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image cluster2 mirror image_41 128M 2023-12-10T19:09:48.673 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:09:48.673 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:09:48.673 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:09:48.673 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:09:48.673 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_41 2023-12-10T19:09:48.673 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:09:48.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local size=128 2023-12-10T19:09:48.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n 128M ] 2023-12-10T19:09:48.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ size=128M 2023-12-10T19:09:48.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:09:48.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_41 2023-12-10T19:09:48.725 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ is_pool_mirror_mode_image mirror 2023-12-10T19:09:48.725 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:09:48.725 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ image = image ] 2023-12-10T19:09:48.725 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:09:48.725 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ enable_mirror cluster2 mirror image_41 snapshot 2023-12-10T19:09:48.725 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:09:48.725 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:09:48.725 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_41 2023-12-10T19:09:48.725 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:09:48.725 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_41 snapshot 2023-12-10T19:09:49.598 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirroring enabled 2023-12-10T19:09:49.610 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 info mirror/image_41 2023-12-10T19:09:49.658 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_41': 2023-12-10T19:09:49.658 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:09:49.658 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:09:49.658 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 1 2023-12-10T19:09:49.658 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 145cca82ed28 2023-12-10T19:09:49.658 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.145cca82ed28 2023-12-10T19:09:49.658 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:09:49.658 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering 2023-12-10T19:09:49.659 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:09:49.659 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:09:49.659 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:09:48 2023 2023-12-10T19:09:49.659 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:09:48 2023 2023-12-10T19:09:49.659 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:09:48 2023 2023-12-10T19:09:49.659 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:09:49.659 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:09:49.659 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: 0caf4caf-9ee1-436f-874d-71291f8008fb 2023-12-10T19:09:49.659 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: true 2023-12-10T19:09:49.662 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T19:09:49.662 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stress_write_image cluster2 mirror image_41 2023-12-10T19:09:49.662 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:09:49.662 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:09:49.663 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_41 2023-12-10T19:09:49.663 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-10T19:09:49.664 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local duration=8 2023-12-10T19:09:49.664 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T19:09:49.664 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_41 --debug-rbd=20 --debug-journaler=20 2023-12-10T19:09:49.898 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:100 IOs, 38955008 bytes 2023-12-10T19:09:50.104 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:200 IOs, 84320256 bytes 2023-12-10T19:09:50.246 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:300 IOs, 94891008 bytes 2023-12-10T19:09:50.426 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:400 IOs, 119325696 bytes 2023-12-10T19:09:50.630 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:500 IOs, 161366016 bytes 2023-12-10T19:09:50.792 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:600 IOs, 188347392 bytes 2023-12-10T19:09:50.932 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:700 IOs, 209807360 bytes 2023-12-10T19:09:51.200 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:800 IOs, 284426240 bytes 2023-12-10T19:09:51.648 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:900 IOs, 343044096 bytes 2023-12-10T19:09:51.827 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1000 IOs, 375229440 bytes 2023-12-10T19:09:52.032 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1100 IOs, 421598208 bytes 2023-12-10T19:09:52.264 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1200 IOs, 492269568 bytes 2023-12-10T19:09:52.525 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1300 IOs, 553307136 bytes 2023-12-10T19:09:52.642 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1400 IOs, 570517504 bytes 2023-12-10T19:09:52.838 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1500 IOs, 591756288 bytes 2023-12-10T19:09:53.097 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1600 IOs, 663366656 bytes 2023-12-10T19:09:53.318 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1700 IOs, 713773056 bytes 2023-12-10T19:09:53.453 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1800 IOs, 726211584 bytes 2023-12-10T19:09:53.693 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1900 IOs, 799210496 bytes 2023-12-10T19:09:53.994 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2000 IOs, 887079936 bytes 2023-12-10T19:09:54.160 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2100 IOs, 910116864 bytes 2023-12-10T19:09:54.315 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2200 IOs, 933724160 bytes 2023-12-10T19:09:54.488 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2300 IOs, 968193024 bytes 2023-12-10T19:09:54.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2400 IOs, 1031349248 bytes 2023-12-10T19:09:54.847 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2500 IOs, 1035719680 bytes 2023-12-10T19:09:55.196 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2600 IOs, 1164457984 bytes 2023-12-10T19:09:55.358 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2700 IOs, 1188483072 bytes 2023-12-10T19:09:55.577 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2800 IOs, 1244281856 bytes 2023-12-10T19:09:55.766 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2900 IOs, 1285963776 bytes 2023-12-10T19:09:55.979 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3000 IOs, 1332076544 bytes 2023-12-10T19:09:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3100 IOs, 1385534464 bytes 2023-12-10T19:09:56.397 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3200 IOs, 1429478400 bytes 2023-12-10T19:09:56.642 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3300 IOs, 1496034304 bytes 2023-12-10T19:09:56.880 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3400 IOs, 1567497216 bytes 2023-12-10T19:09:57.049 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3500 IOs, 1587805184 bytes 2023-12-10T19:09:57.245 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3600 IOs, 1640339456 bytes 2023-12-10T19:09:57.441 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3700 IOs, 1661763584 bytes 2023-12-10T19:09:57.603 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3800 IOs, 1686421504 bytes 2023-12-10T19:09:57.670 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=124 2023-12-10T19:09:57.670 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T19:09:57.671 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 124 -eq 124 ] 2023-12-10T19:09:57.671 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:09:57.671 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_42 2023-12-10T19:09:57.671 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image_and_enable_mirror cluster2 mirror image_42 snapshot 128M 2023-12-10T19:09:57.671 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:09:57.671 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:09:57.671 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:09:57.671 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:09:57.671 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_42 2023-12-10T19:09:57.671 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:09:57.671 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:09:57.671 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n snapshot ] 2023-12-10T19:09:57.671 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:09:57.671 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image cluster2 mirror image_42 128M 2023-12-10T19:09:57.671 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:09:57.672 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:09:57.672 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:09:57.672 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:09:57.672 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_42 2023-12-10T19:09:57.672 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:09:57.672 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local size=128 2023-12-10T19:09:57.672 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n 128M ] 2023-12-10T19:09:57.672 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ size=128M 2023-12-10T19:09:57.672 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:09:57.672 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_42 2023-12-10T19:09:57.720 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ is_pool_mirror_mode_image mirror 2023-12-10T19:09:57.720 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:09:57.720 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ image = image ] 2023-12-10T19:09:57.721 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:09:57.721 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ enable_mirror cluster2 mirror image_42 snapshot 2023-12-10T19:09:57.721 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:09:57.721 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:09:57.721 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_42 2023-12-10T19:09:57.721 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:09:57.721 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_42 snapshot 2023-12-10T19:09:58.587 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirroring enabled 2023-12-10T19:09:58.598 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 info mirror/image_42 2023-12-10T19:09:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_42': 2023-12-10T19:09:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:09:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:09:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 1 2023-12-10T19:09:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 1464d67694bc 2023-12-10T19:09:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.1464d67694bc 2023-12-10T19:09:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:09:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering 2023-12-10T19:09:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:09:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:09:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:09:57 2023 2023-12-10T19:09:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:09:57 2023 2023-12-10T19:09:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:09:57 2023 2023-12-10T19:09:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:09:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:09:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: 472ccd57-dfb6-4e9c-bb2d-657bb53c8829 2023-12-10T19:09:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: true 2023-12-10T19:09:58.649 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T19:09:58.649 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stress_write_image cluster2 mirror image_42 2023-12-10T19:09:58.649 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:09:58.649 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:09:58.649 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_42 2023-12-10T19:09:58.649 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-10T19:09:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local duration=6 2023-12-10T19:09:58.651 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T19:09:58.651 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_42 --debug-rbd=20 --debug-journaler=20 2023-12-10T19:09:58.938 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:100 IOs, 67941376 bytes 2023-12-10T19:09:59.126 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:200 IOs, 105092096 bytes 2023-12-10T19:09:59.325 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:300 IOs, 155083776 bytes 2023-12-10T19:09:59.480 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:400 IOs, 193062912 bytes 2023-12-10T19:09:59.725 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:500 IOs, 245139456 bytes 2023-12-10T19:09:59.898 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:600 IOs, 270755840 bytes 2023-12-10T19:10:00.101 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:700 IOs, 320611328 bytes 2023-12-10T19:10:00.305 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:800 IOs, 363034624 bytes 2023-12-10T19:10:00.488 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:900 IOs, 400067584 bytes 2023-12-10T19:10:00.609 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1000 IOs, 400677888 bytes 2023-12-10T19:10:00.731 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1100 IOs, 401297408 bytes 2023-12-10T19:10:00.915 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1200 IOs, 432860160 bytes 2023-12-10T19:10:01.197 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1300 IOs, 518779904 bytes 2023-12-10T19:10:01.482 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1400 IOs, 591205376 bytes 2023-12-10T19:10:01.729 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1500 IOs, 652591104 bytes 2023-12-10T19:10:01.885 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1600 IOs, 672603136 bytes 2023-12-10T19:10:02.105 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1700 IOs, 730753024 bytes 2023-12-10T19:10:02.276 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1800 IOs, 760648704 bytes 2023-12-10T19:10:02.411 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1900 IOs, 769744896 bytes 2023-12-10T19:10:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2000 IOs, 807757824 bytes 2023-12-10T19:10:02.782 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2100 IOs, 839247872 bytes 2023-12-10T19:10:02.944 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2200 IOs, 865748992 bytes 2023-12-10T19:10:03.104 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2300 IOs, 886483968 bytes 2023-12-10T19:10:03.280 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2400 IOs, 923725824 bytes 2023-12-10T19:10:03.445 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2500 IOs, 950986752 bytes 2023-12-10T19:10:03.726 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2600 IOs, 1033842688 bytes 2023-12-10T19:10:03.967 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2700 IOs, 1092012032 bytes 2023-12-10T19:10:04.220 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2800 IOs, 1166363648 bytes 2023-12-10T19:10:04.439 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2900 IOs, 1213936640 bytes 2023-12-10T19:10:04.628 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3000 IOs, 1252105216 bytes 2023-12-10T19:10:04.657 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=124 2023-12-10T19:10:04.657 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T19:10:04.657 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 124 -eq 124 ] 2023-12-10T19:10:04.657 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:10:04.658 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_43 2023-12-10T19:10:04.658 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image_and_enable_mirror cluster2 mirror image_43 snapshot 128M 2023-12-10T19:10:04.658 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:10:04.658 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:10:04.658 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:10:04.658 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:10:04.658 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_43 2023-12-10T19:10:04.658 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:10:04.658 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:10:04.658 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n snapshot ] 2023-12-10T19:10:04.658 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:10:04.658 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image cluster2 mirror image_43 128M 2023-12-10T19:10:04.659 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:10:04.659 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:10:04.659 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:10:04.659 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:10:04.659 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_43 2023-12-10T19:10:04.659 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:10:04.659 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local size=128 2023-12-10T19:10:04.659 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n 128M ] 2023-12-10T19:10:04.659 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ size=128M 2023-12-10T19:10:04.659 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:10:04.659 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_43 2023-12-10T19:10:04.709 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ is_pool_mirror_mode_image mirror 2023-12-10T19:10:04.709 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:10:04.709 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ image = image ] 2023-12-10T19:10:04.709 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:10:04.709 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ enable_mirror cluster2 mirror image_43 snapshot 2023-12-10T19:10:04.710 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:10:04.710 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:10:04.710 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_43 2023-12-10T19:10:04.710 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:10:04.710 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_43 snapshot 2023-12-10T19:10:05.232 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirroring enabled 2023-12-10T19:10:05.243 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 info mirror/image_43 2023-12-10T19:10:05.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_43': 2023-12-10T19:10:05.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:10:05.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:10:05.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 1 2023-12-10T19:10:05.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 146ca50b31c9 2023-12-10T19:10:05.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.146ca50b31c9 2023-12-10T19:10:05.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:10:05.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering 2023-12-10T19:10:05.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:10:05.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:10:05.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:10:04 2023 2023-12-10T19:10:05.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:10:04 2023 2023-12-10T19:10:05.290 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:10:04 2023 2023-12-10T19:10:05.290 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:10:05.290 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:10:05.290 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: bc83a314-8e55-4363-aaa1-1ab49240de8f 2023-12-10T19:10:05.290 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: true 2023-12-10T19:10:05.293 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T19:10:05.293 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stress_write_image cluster2 mirror image_43 2023-12-10T19:10:05.293 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:10:05.293 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:10:05.293 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_43 2023-12-10T19:10:05.293 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-10T19:10:05.294 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local duration=8 2023-12-10T19:10:05.294 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T19:10:05.294 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_43 --debug-rbd=20 --debug-journaler=20 2023-12-10T19:10:05.537 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:100 IOs, 47464448 bytes 2023-12-10T19:10:05.792 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:200 IOs, 125887488 bytes 2023-12-10T19:10:05.972 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:300 IOs, 158370816 bytes 2023-12-10T19:10:06.230 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:400 IOs, 228375552 bytes 2023-12-10T19:10:06.447 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:500 IOs, 284130304 bytes 2023-12-10T19:10:06.633 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:600 IOs, 313412608 bytes 2023-12-10T19:10:06.774 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:700 IOs, 329882624 bytes 2023-12-10T19:10:06.991 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:800 IOs, 348870656 bytes 2023-12-10T19:10:07.120 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:900 IOs, 351620096 bytes 2023-12-10T19:10:07.420 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1000 IOs, 450291712 bytes 2023-12-10T19:10:07.599 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1100 IOs, 483804160 bytes 2023-12-10T19:10:07.798 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1200 IOs, 519844864 bytes 2023-12-10T19:10:08.001 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1300 IOs, 574194688 bytes 2023-12-10T19:10:08.225 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1400 IOs, 631034880 bytes 2023-12-10T19:10:08.544 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1500 IOs, 691205120 bytes 2023-12-10T19:10:08.725 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1600 IOs, 727897088 bytes 2023-12-10T19:10:08.925 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1700 IOs, 775348224 bytes 2023-12-10T19:10:09.106 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1800 IOs, 816418816 bytes 2023-12-10T19:10:09.273 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1900 IOs, 841325568 bytes 2023-12-10T19:10:09.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2000 IOs, 938331136 bytes 2023-12-10T19:10:09.885 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2100 IOs, 985629696 bytes 2023-12-10T19:10:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2200 IOs, 1008257024 bytes 2023-12-10T19:10:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2300 IOs, 1008876544 bytes 2023-12-10T19:10:10.313 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2400 IOs, 1033539584 bytes 2023-12-10T19:10:10.553 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2500 IOs, 1094382592 bytes 2023-12-10T19:10:10.728 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2600 IOs, 1124663296 bytes 2023-12-10T19:10:10.840 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2700 IOs, 1129393152 bytes 2023-12-10T19:10:11.047 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2800 IOs, 1184061440 bytes 2023-12-10T19:10:11.203 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2900 IOs, 1204168704 bytes 2023-12-10T19:10:11.322 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3000 IOs, 1207006208 bytes 2023-12-10T19:10:11.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3100 IOs, 1285295104 bytes 2023-12-10T19:10:11.754 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3200 IOs, 1310375936 bytes 2023-12-10T19:10:11.973 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3300 IOs, 1364952064 bytes 2023-12-10T19:10:12.090 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3400 IOs, 1367879680 bytes 2023-12-10T19:10:12.300 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3500 IOs, 1431311360 bytes 2023-12-10T19:10:12.469 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3600 IOs, 1458025472 bytes 2023-12-10T19:10:12.654 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3700 IOs, 1499445248 bytes 2023-12-10T19:10:12.912 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3800 IOs, 1569670144 bytes 2023-12-10T19:10:13.072 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3900 IOs, 1595247616 bytes 2023-12-10T19:10:13.203 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4000 IOs, 1604194304 bytes 2023-12-10T19:10:13.299 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=124 2023-12-10T19:10:13.299 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T19:10:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 124 -eq 124 ] 2023-12-10T19:10:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:10:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_44 2023-12-10T19:10:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image_and_enable_mirror cluster2 mirror image_44 snapshot 128M 2023-12-10T19:10:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:10:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:10:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:10:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:10:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_44 2023-12-10T19:10:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:10:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:10:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n snapshot ] 2023-12-10T19:10:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:10:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image cluster2 mirror image_44 128M 2023-12-10T19:10:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:10:13.301 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:10:13.301 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:10:13.301 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:10:13.301 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_44 2023-12-10T19:10:13.301 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:10:13.301 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local size=128 2023-12-10T19:10:13.301 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n 128M ] 2023-12-10T19:10:13.301 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ size=128M 2023-12-10T19:10:13.301 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:10:13.301 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_44 2023-12-10T19:10:13.345 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ is_pool_mirror_mode_image mirror 2023-12-10T19:10:13.346 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:10:13.346 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ image = image ] 2023-12-10T19:10:13.346 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:10:13.346 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ enable_mirror cluster2 mirror image_44 snapshot 2023-12-10T19:10:13.346 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:10:13.346 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:10:13.346 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_44 2023-12-10T19:10:13.346 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:10:13.346 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_44 snapshot 2023-12-10T19:10:13.848 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirroring enabled 2023-12-10T19:10:13.862 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 info mirror/image_44 2023-12-10T19:10:13.909 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_44': 2023-12-10T19:10:13.910 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:10:13.910 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:10:13.910 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 1 2023-12-10T19:10:13.910 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 14757605c69e 2023-12-10T19:10:13.910 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.14757605c69e 2023-12-10T19:10:13.910 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:10:13.910 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering 2023-12-10T19:10:13.910 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:10:13.910 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:10:13.910 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:10:13 2023 2023-12-10T19:10:13.910 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:10:13 2023 2023-12-10T19:10:13.910 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:10:13 2023 2023-12-10T19:10:13.910 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:10:13.910 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:10:13.910 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: e95c6ade-d549-422b-a9cd-187a5dd6a0d6 2023-12-10T19:10:13.911 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: true 2023-12-10T19:10:13.913 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T19:10:13.914 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stress_write_image cluster2 mirror image_44 2023-12-10T19:10:13.914 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:10:13.914 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:10:13.914 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_44 2023-12-10T19:10:13.914 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-10T19:10:13.915 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local duration=6 2023-12-10T19:10:13.915 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T19:10:13.915 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_44 --debug-rbd=20 --debug-journaler=20 2023-12-10T19:10:14.138 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:100 IOs, 35981312 bytes 2023-12-10T19:10:14.398 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:200 IOs, 116252672 bytes 2023-12-10T19:10:14.520 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:300 IOs, 116908032 bytes 2023-12-10T19:10:14.700 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:400 IOs, 161226752 bytes 2023-12-10T19:10:14.956 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:500 IOs, 230150144 bytes 2023-12-10T19:10:15.183 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:600 IOs, 284315648 bytes 2023-12-10T19:10:15.351 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:700 IOs, 320586752 bytes 2023-12-10T19:10:15.585 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:800 IOs, 376463360 bytes 2023-12-10T19:10:15.724 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:900 IOs, 390597632 bytes 2023-12-10T19:10:15.966 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1000 IOs, 440993792 bytes 2023-12-10T19:10:16.102 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1100 IOs, 451348480 bytes 2023-12-10T19:10:16.348 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1200 IOs, 511240192 bytes 2023-12-10T19:10:16.492 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1300 IOs, 531010560 bytes 2023-12-10T19:10:16.718 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1400 IOs, 589072384 bytes 2023-12-10T19:10:16.926 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1500 IOs, 634567680 bytes 2023-12-10T19:10:17.132 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1600 IOs, 674837504 bytes 2023-12-10T19:10:17.273 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1700 IOs, 690120704 bytes 2023-12-10T19:10:17.494 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1800 IOs, 744090624 bytes 2023-12-10T19:10:17.708 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1900 IOs, 793581568 bytes 2023-12-10T19:10:17.875 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2000 IOs, 818365440 bytes 2023-12-10T19:10:18.106 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2100 IOs, 881266688 bytes 2023-12-10T19:10:18.287 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2200 IOs, 902362112 bytes 2023-12-10T19:10:18.429 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2300 IOs, 916637696 bytes 2023-12-10T19:10:18.584 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2400 IOs, 935629824 bytes 2023-12-10T19:10:18.709 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2500 IOs, 936255488 bytes 2023-12-10T19:10:18.855 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2600 IOs, 955334656 bytes 2023-12-10T19:10:18.964 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2700 IOs, 958586880 bytes 2023-12-10T19:10:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2800 IOs, 1002093568 bytes 2023-12-10T19:10:19.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2900 IOs, 1032956928 bytes 2023-12-10T19:10:19.527 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3000 IOs, 1067949056 bytes 2023-12-10T19:10:19.845 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3100 IOs, 1175190528 bytes 2023-12-10T19:10:19.921 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=124 2023-12-10T19:10:19.921 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T19:10:19.922 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 124 -eq 124 ] 2023-12-10T19:10:19.922 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:10:19.922 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_45 2023-12-10T19:10:19.922 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image_and_enable_mirror cluster2 mirror image_45 snapshot 128M 2023-12-10T19:10:19.922 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:10:19.922 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:10:19.922 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:10:19.922 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:10:19.922 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_45 2023-12-10T19:10:19.922 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:10:19.922 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:10:19.922 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n snapshot ] 2023-12-10T19:10:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:10:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image cluster2 mirror image_45 128M 2023-12-10T19:10:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:10:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:10:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:10:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:10:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_45 2023-12-10T19:10:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:10:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local size=128 2023-12-10T19:10:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n 128M ] 2023-12-10T19:10:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ size=128M 2023-12-10T19:10:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:10:19.924 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_45 2023-12-10T19:10:19.968 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ is_pool_mirror_mode_image mirror 2023-12-10T19:10:19.968 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:10:19.968 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ image = image ] 2023-12-10T19:10:19.968 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:10:19.968 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ enable_mirror cluster2 mirror image_45 snapshot 2023-12-10T19:10:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:10:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:10:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_45 2023-12-10T19:10:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:10:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_45 snapshot 2023-12-10T19:10:20.656 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirroring enabled 2023-12-10T19:10:20.668 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 info mirror/image_45 2023-12-10T19:10:20.716 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_45': 2023-12-10T19:10:20.716 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:10:20.716 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:10:20.716 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 1 2023-12-10T19:10:20.716 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 147fbbb91e4b 2023-12-10T19:10:20.716 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.147fbbb91e4b 2023-12-10T19:10:20.717 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:10:20.717 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering 2023-12-10T19:10:20.717 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:10:20.717 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:10:20.717 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:10:19 2023 2023-12-10T19:10:20.717 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:10:19 2023 2023-12-10T19:10:20.717 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:10:19 2023 2023-12-10T19:10:20.717 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:10:20.717 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:10:20.717 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: 40a91520-4e7a-4ba5-aaf0-64312a2dcb5a 2023-12-10T19:10:20.717 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: true 2023-12-10T19:10:20.720 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T19:10:20.721 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stress_write_image cluster2 mirror image_45 2023-12-10T19:10:20.721 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:10:20.721 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:10:20.721 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_45 2023-12-10T19:10:20.721 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-10T19:10:20.722 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local duration=11 2023-12-10T19:10:20.722 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T19:10:20.722 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_45 --debug-rbd=20 --debug-journaler=20 2023-12-10T19:10:21.096 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:100 IOs, 102986752 bytes 2023-12-10T19:10:21.210 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:200 IOs, 123550720 bytes 2023-12-10T19:10:21.389 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:300 IOs, 148890624 bytes 2023-12-10T19:10:21.651 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:400 IOs, 223273984 bytes 2023-12-10T19:10:21.882 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:500 IOs, 274270208 bytes 2023-12-10T19:10:22.119 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:600 IOs, 343205888 bytes 2023-12-10T19:10:22.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:700 IOs, 379403264 bytes 2023-12-10T19:10:22.509 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:800 IOs, 414200832 bytes 2023-12-10T19:10:22.677 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:900 IOs, 441509888 bytes 2023-12-10T19:10:22.795 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1000 IOs, 453844992 bytes 2023-12-10T19:10:22.998 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1100 IOs, 502330368 bytes 2023-12-10T19:10:23.207 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1200 IOs, 536458240 bytes 2023-12-10T19:10:23.374 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1300 IOs, 568274944 bytes 2023-12-10T19:10:23.574 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1400 IOs, 617182208 bytes 2023-12-10T19:10:23.743 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1500 IOs, 655441920 bytes 2023-12-10T19:10:23.946 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1600 IOs, 690212864 bytes 2023-12-10T19:10:24.201 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1700 IOs, 769188864 bytes 2023-12-10T19:10:24.421 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1800 IOs, 823515136 bytes 2023-12-10T19:10:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1900 IOs, 844072960 bytes 2023-12-10T19:10:24.877 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2000 IOs, 936434688 bytes 2023-12-10T19:10:25.032 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2100 IOs, 955524096 bytes 2023-12-10T19:10:25.276 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2200 IOs, 975405056 bytes 2023-12-10T19:10:25.411 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2300 IOs, 981560320 bytes 2023-12-10T19:10:25.576 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2400 IOs, 1013258240 bytes 2023-12-10T19:10:25.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2500 IOs, 1038628864 bytes 2023-12-10T19:10:25.850 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2600 IOs, 1048080384 bytes 2023-12-10T19:10:26.067 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2700 IOs, 1103355904 bytes 2023-12-10T19:10:26.223 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2800 IOs, 1133333504 bytes 2023-12-10T19:10:26.406 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2900 IOs, 1166093312 bytes 2023-12-10T19:10:26.542 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3000 IOs, 1194975232 bytes 2023-12-10T19:10:26.796 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3100 IOs, 1253180416 bytes 2023-12-10T19:10:27.030 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3200 IOs, 1313558528 bytes 2023-12-10T19:10:27.202 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3300 IOs, 1342328832 bytes 2023-12-10T19:10:27.350 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3400 IOs, 1362098176 bytes 2023-12-10T19:10:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3500 IOs, 1381957632 bytes 2023-12-10T19:10:27.650 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3600 IOs, 1404992512 bytes 2023-12-10T19:10:27.963 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3700 IOs, 1515044864 bytes 2023-12-10T19:10:28.207 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3800 IOs, 1566219264 bytes 2023-12-10T19:10:28.353 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3900 IOs, 1585102848 bytes 2023-12-10T19:10:28.561 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4000 IOs, 1644021760 bytes 2023-12-10T19:10:28.828 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4100 IOs, 1718893568 bytes 2023-12-10T19:10:28.998 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4200 IOs, 1744395264 bytes 2023-12-10T19:10:29.114 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4300 IOs, 1744989184 bytes 2023-12-10T19:10:29.281 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4400 IOs, 1775989760 bytes 2023-12-10T19:10:29.505 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4500 IOs, 1823035392 bytes 2023-12-10T19:10:29.638 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4600 IOs, 1860404224 bytes 2023-12-10T19:10:29.798 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4700 IOs, 1861019648 bytes 2023-12-10T19:10:30.014 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4800 IOs, 1925797888 bytes 2023-12-10T19:10:30.163 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4900 IOs, 1933766656 bytes 2023-12-10T19:10:30.313 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5000 IOs, 1953573888 bytes 2023-12-10T19:10:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5100 IOs, 1954208768 bytes 2023-12-10T19:10:30.690 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5200 IOs, 2034518016 bytes 2023-12-10T19:10:30.821 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5300 IOs, 2045174784 bytes 2023-12-10T19:10:31.039 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5400 IOs, 2105807872 bytes 2023-12-10T19:10:31.278 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5500 IOs, 2160248832 bytes 2023-12-10T19:10:31.604 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5600 IOs, 2272852992 bytes 2023-12-10T19:10:31.729 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=124 2023-12-10T19:10:31.729 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T19:10:31.729 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 124 -eq 124 ] 2023-12-10T19:10:31.729 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:10:31.729 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_46 2023-12-10T19:10:31.729 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image_and_enable_mirror cluster2 mirror image_46 snapshot 128M 2023-12-10T19:10:31.729 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:10:31.729 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:10:31.729 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:10:31.729 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:10:31.729 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_46 2023-12-10T19:10:31.729 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:10:31.729 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:10:31.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n snapshot ] 2023-12-10T19:10:31.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:10:31.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image cluster2 mirror image_46 128M 2023-12-10T19:10:31.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:10:31.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:10:31.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:10:31.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:10:31.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_46 2023-12-10T19:10:31.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:10:31.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local size=128 2023-12-10T19:10:31.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n 128M ] 2023-12-10T19:10:31.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ size=128M 2023-12-10T19:10:31.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:10:31.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_46 2023-12-10T19:10:31.773 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ is_pool_mirror_mode_image mirror 2023-12-10T19:10:31.773 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:10:31.774 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ image = image ] 2023-12-10T19:10:31.774 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:10:31.774 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ enable_mirror cluster2 mirror image_46 snapshot 2023-12-10T19:10:31.774 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:10:31.774 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:10:31.774 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_46 2023-12-10T19:10:31.774 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:10:31.774 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_46 snapshot 2023-12-10T19:10:32.655 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirroring enabled 2023-12-10T19:10:32.669 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 info mirror/image_46 2023-12-10T19:10:32.717 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_46': 2023-12-10T19:10:32.717 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:10:32.717 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:10:32.717 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 1 2023-12-10T19:10:32.717 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 1487b1c92a3a 2023-12-10T19:10:32.717 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.1487b1c92a3a 2023-12-10T19:10:32.717 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:10:32.717 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering 2023-12-10T19:10:32.717 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:10:32.718 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:10:32.718 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:10:31 2023 2023-12-10T19:10:32.718 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:10:31 2023 2023-12-10T19:10:32.718 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:10:31 2023 2023-12-10T19:10:32.718 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:10:32.718 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:10:32.718 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: 509368f6-035e-4be6-926a-dc4bce662e88 2023-12-10T19:10:32.718 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: true 2023-12-10T19:10:32.721 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T19:10:32.721 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stress_write_image cluster2 mirror image_46 2023-12-10T19:10:32.721 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:10:32.721 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:10:32.721 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_46 2023-12-10T19:10:32.721 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-10T19:10:32.724 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local duration=7 2023-12-10T19:10:32.724 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T19:10:32.724 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_46 --debug-rbd=20 --debug-journaler=20 2023-12-10T19:10:32.984 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:100 IOs, 56010752 bytes 2023-12-10T19:10:33.161 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:200 IOs, 92539904 bytes 2023-12-10T19:10:33.332 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:300 IOs, 137440256 bytes 2023-12-10T19:10:33.568 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:400 IOs, 202680320 bytes 2023-12-10T19:10:33.756 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:500 IOs, 224948224 bytes 2023-12-10T19:10:33.883 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:600 IOs, 230523904 bytes 2023-12-10T19:10:34.034 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:700 IOs, 252773376 bytes 2023-12-10T19:10:34.301 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:800 IOs, 330480640 bytes 2023-12-10T19:10:34.462 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:900 IOs, 377205760 bytes 2023-12-10T19:10:34.647 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1000 IOs, 411893760 bytes 2023-12-10T19:10:34.955 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1100 IOs, 492206080 bytes 2023-12-10T19:10:35.188 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1200 IOs, 547237888 bytes 2023-12-10T19:10:35.337 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1300 IOs, 561623040 bytes 2023-12-10T19:10:35.562 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1400 IOs, 615155712 bytes 2023-12-10T19:10:35.733 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1500 IOs, 638516224 bytes 2023-12-10T19:10:35.871 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1600 IOs, 653757440 bytes 2023-12-10T19:10:36.122 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1700 IOs, 705286144 bytes 2023-12-10T19:10:36.283 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1800 IOs, 742219776 bytes 2023-12-10T19:10:36.532 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1900 IOs, 793712640 bytes 2023-12-10T19:10:36.799 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2000 IOs, 855813120 bytes 2023-12-10T19:10:36.942 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2100 IOs, 868793344 bytes 2023-12-10T19:10:37.060 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2200 IOs, 869401600 bytes 2023-12-10T19:10:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2300 IOs, 897768448 bytes 2023-12-10T19:10:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2400 IOs, 918755328 bytes 2023-12-10T19:10:37.553 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2500 IOs, 951141376 bytes 2023-12-10T19:10:37.693 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2600 IOs, 967244800 bytes 2023-12-10T19:10:38.013 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2700 IOs, 1053507584 bytes 2023-12-10T19:10:38.168 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2800 IOs, 1070499840 bytes 2023-12-10T19:10:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2900 IOs, 1131354112 bytes 2023-12-10T19:10:38.566 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3000 IOs, 1152925696 bytes 2023-12-10T19:10:38.762 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3100 IOs, 1193979904 bytes 2023-12-10T19:10:39.034 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3200 IOs, 1253657600 bytes 2023-12-10T19:10:39.291 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3300 IOs, 1341986816 bytes 2023-12-10T19:10:39.481 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3400 IOs, 1358478336 bytes 2023-12-10T19:10:39.658 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3500 IOs, 1420983296 bytes 2023-12-10T19:10:39.731 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=124 2023-12-10T19:10:39.731 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T19:10:39.731 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 124 -eq 124 ] 2023-12-10T19:10:39.731 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:10:39.731 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_47 2023-12-10T19:10:39.731 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image_and_enable_mirror cluster2 mirror image_47 snapshot 128M 2023-12-10T19:10:39.731 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:10:39.731 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:10:39.731 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:10:39.731 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:10:39.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_47 2023-12-10T19:10:39.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:10:39.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:10:39.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n snapshot ] 2023-12-10T19:10:39.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:10:39.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image cluster2 mirror image_47 128M 2023-12-10T19:10:39.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:10:39.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:10:39.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:10:39.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:10:39.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_47 2023-12-10T19:10:39.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:10:39.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local size=128 2023-12-10T19:10:39.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n 128M ] 2023-12-10T19:10:39.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ size=128M 2023-12-10T19:10:39.733 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:10:39.733 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_47 2023-12-10T19:10:39.779 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ is_pool_mirror_mode_image mirror 2023-12-10T19:10:39.779 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:10:39.780 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ image = image ] 2023-12-10T19:10:39.780 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:10:39.780 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ enable_mirror cluster2 mirror image_47 snapshot 2023-12-10T19:10:39.780 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:10:39.780 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:10:39.780 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_47 2023-12-10T19:10:39.780 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:10:39.780 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_47 snapshot 2023-12-10T19:10:40.614 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirroring enabled 2023-12-10T19:10:40.625 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 info mirror/image_47 2023-12-10T19:10:40.673 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_47': 2023-12-10T19:10:40.673 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:10:40.673 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:10:40.673 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 1 2023-12-10T19:10:40.673 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 1491c14645be 2023-12-10T19:10:40.673 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.1491c14645be 2023-12-10T19:10:40.673 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:10:40.673 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering 2023-12-10T19:10:40.673 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:10:40.673 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:10:40.673 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:10:39 2023 2023-12-10T19:10:40.673 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:10:39 2023 2023-12-10T19:10:40.673 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:10:39 2023 2023-12-10T19:10:40.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:10:40.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:10:40.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: 35463a96-219e-457b-aaa2-28a56599112e 2023-12-10T19:10:40.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: true 2023-12-10T19:10:40.677 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T19:10:40.677 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stress_write_image cluster2 mirror image_47 2023-12-10T19:10:40.677 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:10:40.677 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:10:40.677 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_47 2023-12-10T19:10:40.677 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-10T19:10:40.679 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local duration=13 2023-12-10T19:10:40.679 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T19:10:40.679 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_47 --debug-rbd=20 --debug-journaler=20 2023-12-10T19:10:40.900 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:100 IOs, 23550976 bytes 2023-12-10T19:10:41.137 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:200 IOs, 85500928 bytes 2023-12-10T19:10:41.361 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:300 IOs, 144011264 bytes 2023-12-10T19:10:41.569 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:400 IOs, 188063744 bytes 2023-12-10T19:10:41.771 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:500 IOs, 237834240 bytes 2023-12-10T19:10:42.018 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:600 IOs, 307214336 bytes 2023-12-10T19:10:42.134 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:700 IOs, 307798016 bytes 2023-12-10T19:10:42.375 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:800 IOs, 381401088 bytes 2023-12-10T19:10:42.547 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:900 IOs, 404598784 bytes 2023-12-10T19:10:42.759 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1000 IOs, 456111104 bytes 2023-12-10T19:10:42.935 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1100 IOs, 485976064 bytes 2023-12-10T19:10:43.046 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1200 IOs, 488213504 bytes 2023-12-10T19:10:43.172 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1300 IOs, 493883392 bytes 2023-12-10T19:10:43.413 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1400 IOs, 559321088 bytes 2023-12-10T19:10:43.524 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1500 IOs, 559941632 bytes 2023-12-10T19:10:43.718 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1600 IOs, 608192512 bytes 2023-12-10T19:10:43.955 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1700 IOs, 666089472 bytes 2023-12-10T19:10:44.188 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1800 IOs, 720470016 bytes 2023-12-10T19:10:44.406 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1900 IOs, 775312384 bytes 2023-12-10T19:10:44.782 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2000 IOs, 922454016 bytes 2023-12-10T19:10:44.935 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2100 IOs, 937187328 bytes 2023-12-10T19:10:45.094 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2200 IOs, 961357824 bytes 2023-12-10T19:10:45.271 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2300 IOs, 990497792 bytes 2023-12-10T19:10:45.414 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2400 IOs, 1025347584 bytes 2023-12-10T19:10:45.634 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2500 IOs, 1067600896 bytes 2023-12-10T19:10:45.783 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2600 IOs, 1081114624 bytes 2023-12-10T19:10:45.943 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2700 IOs, 1100000256 bytes 2023-12-10T19:10:46.244 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2800 IOs, 1207406592 bytes 2023-12-10T19:10:46.426 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2900 IOs, 1224095744 bytes 2023-12-10T19:10:46.562 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3000 IOs, 1232009216 bytes 2023-12-10T19:10:46.705 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3100 IOs, 1254454272 bytes 2023-12-10T19:10:47.002 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3200 IOs, 1340988416 bytes 2023-12-10T19:10:47.243 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3300 IOs, 1403856896 bytes 2023-12-10T19:10:47.537 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3400 IOs, 1489326080 bytes 2023-12-10T19:10:47.740 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3500 IOs, 1536988160 bytes 2023-12-10T19:10:47.861 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3600 IOs, 1537588224 bytes 2023-12-10T19:10:48.060 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3700 IOs, 1616817152 bytes 2023-12-10T19:10:48.269 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3800 IOs, 1644804096 bytes 2023-12-10T19:10:48.465 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3900 IOs, 1697967104 bytes 2023-12-10T19:10:48.716 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4000 IOs, 1778369536 bytes 2023-12-10T19:10:48.840 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4100 IOs, 1779005440 bytes 2023-12-10T19:10:49.060 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4200 IOs, 1853046784 bytes 2023-12-10T19:10:49.294 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4300 IOs, 1901854720 bytes 2023-12-10T19:10:49.454 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4400 IOs, 1926937600 bytes 2023-12-10T19:10:49.561 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4500 IOs, 1927556096 bytes 2023-12-10T19:10:49.708 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4600 IOs, 1942514688 bytes 2023-12-10T19:10:49.852 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6849189624551398) > weight (0.25) 2023-12-10T19:10:49.852 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.4219583405809644) > weight (0.25) 2023-12-10T19:10:49.852 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2023-12-10T19:10:49.853 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2023-12-10T19:10:49.853 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2023-12-10T19:10:49.854 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi076.stderr:2023-12-10T19:10:49.861+0000 7f51504be640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 52065) UID: 1000 2023-12-10T19:10:49.907 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4700 IOs, 1979979776 bytes 2023-12-10T19:10:50.085 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4800 IOs, 2011267072 bytes 2023-12-10T19:10:50.342 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4900 IOs, 2084525056 bytes 2023-12-10T19:10:50.505 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5000 IOs, 2108386304 bytes 2023-12-10T19:10:50.777 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5100 IOs, 2192599040 bytes 2023-12-10T19:10:50.970 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5200 IOs, 2238879744 bytes 2023-12-10T19:10:51.196 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5300 IOs, 2287976448 bytes 2023-12-10T19:10:51.436 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5400 IOs, 2349370368 bytes 2023-12-10T19:10:51.650 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5500 IOs, 2407424000 bytes 2023-12-10T19:10:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5600 IOs, 2423647232 bytes 2023-12-10T19:10:51.854 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.540946940575394) > weight (0.25) 2023-12-10T19:10:51.854 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.5532202845026372) > weight (0.25) 2023-12-10T19:10:51.854 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.52788023816379) > weight (0.25) 2023-12-10T19:10:51.854 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2023-12-10T19:10:51.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2023-12-10T19:10:51.855 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2023-12-10T19:10:51.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi076.stderr:2023-12-10T19:10:51.860+0000 7f432897c640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 39906) UID: 1000 2023-12-10T19:10:51.960 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5700 IOs, 2446209024 bytes 2023-12-10T19:10:52.118 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5800 IOs, 2474556416 bytes 2023-12-10T19:10:52.278 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5900 IOs, 2500186112 bytes 2023-12-10T19:10:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6000 IOs, 2543666176 bytes 2023-12-10T19:10:52.610 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6100 IOs, 2557659136 bytes 2023-12-10T19:10:52.777 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6200 IOs, 2585099264 bytes 2023-12-10T19:10:52.947 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6300 IOs, 2615106560 bytes 2023-12-10T19:10:53.230 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6400 IOs, 2693714944 bytes 2023-12-10T19:10:53.346 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6500 IOs, 2695616512 bytes 2023-12-10T19:10:53.591 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6600 IOs, 2762670080 bytes 2023-12-10T19:10:53.685 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=124 2023-12-10T19:10:53.685 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T19:10:53.685 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 124 -eq 124 ] 2023-12-10T19:10:53.685 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:10:53.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_48 2023-12-10T19:10:53.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image_and_enable_mirror cluster2 mirror image_48 snapshot 128M 2023-12-10T19:10:53.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:10:53.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:10:53.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:10:53.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:10:53.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_48 2023-12-10T19:10:53.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:10:53.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:10:53.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n snapshot ] 2023-12-10T19:10:53.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:10:53.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image cluster2 mirror image_48 128M 2023-12-10T19:10:53.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:10:53.687 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:10:53.687 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:10:53.687 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:10:53.687 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_48 2023-12-10T19:10:53.687 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:10:53.687 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local size=128 2023-12-10T19:10:53.687 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n 128M ] 2023-12-10T19:10:53.687 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ size=128M 2023-12-10T19:10:53.687 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:10:53.687 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_48 2023-12-10T19:10:53.735 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ is_pool_mirror_mode_image mirror 2023-12-10T19:10:53.735 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:10:53.736 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ image = image ] 2023-12-10T19:10:53.736 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:10:53.736 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ enable_mirror cluster2 mirror image_48 snapshot 2023-12-10T19:10:53.736 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:10:53.736 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:10:53.736 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_48 2023-12-10T19:10:53.736 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:10:53.736 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_48 snapshot 2023-12-10T19:10:54.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirroring enabled 2023-12-10T19:10:54.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 info mirror/image_48 2023-12-10T19:10:54.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_48': 2023-12-10T19:10:54.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:10:54.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:10:54.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 1 2023-12-10T19:10:54.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 149a7661c1af 2023-12-10T19:10:54.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.149a7661c1af 2023-12-10T19:10:54.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:10:54.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering 2023-12-10T19:10:54.731 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:10:54.731 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:10:54.731 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:10:53 2023 2023-12-10T19:10:54.731 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:10:53 2023 2023-12-10T19:10:54.731 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:10:53 2023 2023-12-10T19:10:54.731 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:10:54.731 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:10:54.731 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: 935218e9-f0e7-416b-a035-66ab9afc74f5 2023-12-10T19:10:54.731 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: true 2023-12-10T19:10:54.734 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T19:10:54.734 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stress_write_image cluster2 mirror image_48 2023-12-10T19:10:54.734 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:10:54.734 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:10:54.734 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_48 2023-12-10T19:10:54.734 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-10T19:10:54.735 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local duration=10 2023-12-10T19:10:54.736 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T19:10:54.736 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_48 --debug-rbd=20 --debug-journaler=20 2023-12-10T19:10:54.853 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2023-12-10T19:10:54.854 INFO:teuthology.orchestra.run:waiting for 600 2023-12-10T19:10:54.854 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2023-12-10T19:10:54.854 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2023-12-10T19:10:54.854 DEBUG:teuthology.orchestra.run.smithi076:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2023-12-10T19:10:54.857 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2023-12-10T19:10:54.857 INFO:tasks.rbd_mirror_thrash:waiting for 118 secs before thrashing 2023-12-10T19:10:54.934 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:100 IOs, 21552128 bytes 2023-12-10T19:10:55.162 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:200 IOs, 68889600 bytes 2023-12-10T19:10:55.393 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:300 IOs, 155611136 bytes 2023-12-10T19:10:55.766 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:400 IOs, 243060736 bytes 2023-12-10T19:10:55.981 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:500 IOs, 297402368 bytes 2023-12-10T19:10:56.164 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:600 IOs, 331856896 bytes 2023-12-10T19:10:56.357 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:700 IOs, 370173952 bytes 2023-12-10T19:10:56.549 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:800 IOs, 428679168 bytes 2023-12-10T19:10:56.720 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:900 IOs, 445333504 bytes 2023-12-10T19:10:56.856 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2023-12-10T19:10:56.856 INFO:teuthology.orchestra.run:waiting for 600 2023-12-10T19:10:56.856 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2023-12-10T19:10:56.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2023-12-10T19:10:56.856 DEBUG:teuthology.orchestra.run.smithi076:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2023-12-10T19:10:56.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2023-12-10T19:10:56.859 INFO:tasks.rbd_mirror_thrash:waiting for 85 secs before thrashing 2023-12-10T19:10:56.944 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1000 IOs, 501383168 bytes 2023-12-10T19:10:57.189 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1100 IOs, 562440192 bytes 2023-12-10T19:10:57.353 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1200 IOs, 583163904 bytes 2023-12-10T19:10:57.551 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1300 IOs, 625527808 bytes 2023-12-10T19:10:57.749 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1400 IOs, 679329792 bytes 2023-12-10T19:10:57.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1500 IOs, 731624448 bytes 2023-12-10T19:10:58.151 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1600 IOs, 757338112 bytes 2023-12-10T19:10:58.311 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1700 IOs, 781706240 bytes 2023-12-10T19:10:58.516 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1800 IOs, 828082176 bytes 2023-12-10T19:10:58.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1900 IOs, 884819968 bytes 2023-12-10T19:10:58.933 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2000 IOs, 929929216 bytes 2023-12-10T19:10:59.165 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2100 IOs, 1001252864 bytes 2023-12-10T19:10:59.298 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2200 IOs, 1001894912 bytes 2023-12-10T19:10:59.498 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2300 IOs, 1048665088 bytes 2023-12-10T19:10:59.638 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2400 IOs, 1063319552 bytes 2023-12-10T19:10:59.753 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2500 IOs, 1067719680 bytes 2023-12-10T19:10:59.903 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2600 IOs, 1100068864 bytes 2023-12-10T19:11:00.223 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2700 IOs, 1197763584 bytes 2023-12-10T19:11:00.587 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2800 IOs, 1320489984 bytes 2023-12-10T19:11:00.846 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2900 IOs, 1396793344 bytes 2023-12-10T19:11:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3000 IOs, 1468049408 bytes 2023-12-10T19:11:01.262 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3100 IOs, 1490825216 bytes 2023-12-10T19:11:01.452 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3200 IOs, 1528066048 bytes 2023-12-10T19:11:01.671 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3300 IOs, 1583208448 bytes 2023-12-10T19:11:01.929 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3400 IOs, 1636516864 bytes 2023-12-10T19:11:02.254 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3500 IOs, 1745393664 bytes 2023-12-10T19:11:02.480 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3600 IOs, 1802553344 bytes 2023-12-10T19:11:02.651 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3700 IOs, 1830632448 bytes 2023-12-10T19:11:02.821 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3800 IOs, 1857577984 bytes 2023-12-10T19:11:02.930 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3900 IOs, 1858196480 bytes 2023-12-10T19:11:03.142 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4000 IOs, 1913460736 bytes 2023-12-10T19:11:03.310 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4100 IOs, 1938437120 bytes 2023-12-10T19:11:03.524 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4200 IOs, 1989452800 bytes 2023-12-10T19:11:03.696 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4300 IOs, 2013107200 bytes 2023-12-10T19:11:03.893 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4400 IOs, 2065689600 bytes 2023-12-10T19:11:04.159 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4500 IOs, 2124421120 bytes 2023-12-10T19:11:04.306 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4600 IOs, 2143892480 bytes 2023-12-10T19:11:04.540 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4700 IOs, 2202091520 bytes 2023-12-10T19:11:04.715 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4800 IOs, 2244851712 bytes 2023-12-10T19:11:04.740 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=124 2023-12-10T19:11:04.740 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T19:11:04.740 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 124 -eq 124 ] 2023-12-10T19:11:04.740 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:11:04.740 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_49 2023-12-10T19:11:04.740 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image_and_enable_mirror cluster2 mirror image_49 snapshot 128M 2023-12-10T19:11:04.740 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:11:04.740 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:11:04.740 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:11:04.740 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:11:04.740 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_49 2023-12-10T19:11:04.740 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:11:04.741 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:11:04.741 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n snapshot ] 2023-12-10T19:11:04.741 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:11:04.741 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image cluster2 mirror image_49 128M 2023-12-10T19:11:04.741 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:11:04.741 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:11:04.741 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:11:04.741 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:11:04.741 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_49 2023-12-10T19:11:04.741 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:11:04.741 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local size=128 2023-12-10T19:11:04.741 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n 128M ] 2023-12-10T19:11:04.741 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ size=128M 2023-12-10T19:11:04.741 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:11:04.741 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_49 2023-12-10T19:11:04.787 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ is_pool_mirror_mode_image mirror 2023-12-10T19:11:04.787 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:11:04.787 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ image = image ] 2023-12-10T19:11:04.787 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:11:04.787 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ enable_mirror cluster2 mirror image_49 snapshot 2023-12-10T19:11:04.787 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:11:04.788 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:11:04.788 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_49 2023-12-10T19:11:04.788 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:11:04.788 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_49 snapshot 2023-12-10T19:11:05.268 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirroring enabled 2023-12-10T19:11:05.281 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 info mirror/image_49 2023-12-10T19:11:05.327 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_49': 2023-12-10T19:11:05.327 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:11:05.328 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:11:05.328 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 1 2023-12-10T19:11:05.328 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 14b11bb4adae 2023-12-10T19:11:05.328 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.14b11bb4adae 2023-12-10T19:11:05.328 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:11:05.328 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering 2023-12-10T19:11:05.328 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:11:05.328 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:11:05.328 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:11:04 2023 2023-12-10T19:11:05.328 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:11:04 2023 2023-12-10T19:11:05.328 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:11:04 2023 2023-12-10T19:11:05.328 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:11:05.328 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:11:05.328 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: cfaa6e78-26bd-4924-bba0-a389e00cd9fc 2023-12-10T19:11:05.328 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: true 2023-12-10T19:11:05.332 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T19:11:05.332 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stress_write_image cluster2 mirror image_49 2023-12-10T19:11:05.332 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:11:05.332 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:11:05.332 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_49 2023-12-10T19:11:05.332 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-10T19:11:05.333 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local duration=8 2023-12-10T19:11:05.333 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T19:11:05.333 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_49 --debug-rbd=20 --debug-journaler=20 2023-12-10T19:11:05.508 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:100 IOs, 8040448 bytes 2023-12-10T19:11:05.700 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:200 IOs, 45573120 bytes 2023-12-10T19:11:05.911 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:300 IOs, 98589696 bytes 2023-12-10T19:11:06.206 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:400 IOs, 196205568 bytes 2023-12-10T19:11:06.350 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:500 IOs, 213069824 bytes 2023-12-10T19:11:06.580 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:600 IOs, 277021696 bytes 2023-12-10T19:11:06.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:700 IOs, 321481728 bytes 2023-12-10T19:11:06.932 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:800 IOs, 329431040 bytes 2023-12-10T19:11:07.141 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:900 IOs, 343186432 bytes 2023-12-10T19:11:07.357 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1000 IOs, 385704960 bytes 2023-12-10T19:11:07.517 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1100 IOs, 406259712 bytes 2023-12-10T19:11:07.759 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1200 IOs, 459710464 bytes 2023-12-10T19:11:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1300 IOs, 523906048 bytes 2023-12-10T19:11:08.158 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1400 IOs, 539898880 bytes 2023-12-10T19:11:08.299 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1500 IOs, 557768704 bytes 2023-12-10T19:11:08.531 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1600 IOs, 614703104 bytes 2023-12-10T19:11:08.659 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1700 IOs, 621119488 bytes 2023-12-10T19:11:08.805 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1800 IOs, 639755264 bytes 2023-12-10T19:11:08.953 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1900 IOs, 664957952 bytes 2023-12-10T19:11:09.101 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2000 IOs, 683940864 bytes 2023-12-10T19:11:09.219 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2100 IOs, 687078400 bytes 2023-12-10T19:11:09.398 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2200 IOs, 728993792 bytes 2023-12-10T19:11:09.579 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2300 IOs, 767182848 bytes 2023-12-10T19:11:09.747 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2400 IOs, 794710016 bytes 2023-12-10T19:11:09.928 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2500 IOs, 852679680 bytes 2023-12-10T19:11:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2600 IOs, 876802048 bytes 2023-12-10T19:11:10.286 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2700 IOs, 895039488 bytes 2023-12-10T19:11:10.484 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2800 IOs, 938118144 bytes 2023-12-10T19:11:10.684 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2900 IOs, 983187456 bytes 2023-12-10T19:11:10.914 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3000 IOs, 1040793600 bytes 2023-12-10T19:11:11.111 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3100 IOs, 1090654208 bytes 2023-12-10T19:11:11.295 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3200 IOs, 1123658752 bytes 2023-12-10T19:11:11.477 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3300 IOs, 1168093184 bytes 2023-12-10T19:11:11.715 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3400 IOs, 1216588800 bytes 2023-12-10T19:11:11.854 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3500 IOs, 1222993920 bytes 2023-12-10T19:11:12.031 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3600 IOs, 1256466432 bytes 2023-12-10T19:11:12.274 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3700 IOs, 1309365248 bytes 2023-12-10T19:11:12.458 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3800 IOs, 1342259200 bytes 2023-12-10T19:11:12.605 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3900 IOs, 1359901696 bytes 2023-12-10T19:11:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4000 IOs, 1372184576 bytes 2023-12-10T19:11:13.050 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4100 IOs, 1457646592 bytes 2023-12-10T19:11:13.233 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4200 IOs, 1478004736 bytes 2023-12-10T19:11:13.339 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=124 2023-12-10T19:11:13.339 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T19:11:13.339 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 124 -eq 124 ] 2023-12-10T19:11:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:11:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_50 2023-12-10T19:11:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image_and_enable_mirror cluster2 mirror image_50 snapshot 128M 2023-12-10T19:11:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:11:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:11:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:11:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:11:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_50 2023-12-10T19:11:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:11:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:11:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n snapshot ] 2023-12-10T19:11:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:11:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image cluster2 mirror image_50 128M 2023-12-10T19:11:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:11:13.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:11:13.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:11:13.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:11:13.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_50 2023-12-10T19:11:13.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:11:13.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local size=128 2023-12-10T19:11:13.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n 128M ] 2023-12-10T19:11:13.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ size=128M 2023-12-10T19:11:13.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:11:13.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_50 2023-12-10T19:11:13.386 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ is_pool_mirror_mode_image mirror 2023-12-10T19:11:13.386 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:11:13.386 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ image = image ] 2023-12-10T19:11:13.386 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:11:13.386 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ enable_mirror cluster2 mirror image_50 snapshot 2023-12-10T19:11:13.387 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:11:13.387 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:11:13.387 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_50 2023-12-10T19:11:13.387 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:11:13.387 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_50 snapshot 2023-12-10T19:11:13.651 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirroring enabled 2023-12-10T19:11:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 info mirror/image_50 2023-12-10T19:11:13.705 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_50': 2023-12-10T19:11:13.705 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:11:13.705 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:11:13.705 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 1 2023-12-10T19:11:13.705 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 14b9a113ca6b 2023-12-10T19:11:13.706 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.14b9a113ca6b 2023-12-10T19:11:13.706 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:11:13.706 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering 2023-12-10T19:11:13.706 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:11:13.706 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:11:13.706 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:11:13 2023 2023-12-10T19:11:13.706 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:11:13 2023 2023-12-10T19:11:13.706 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:11:13 2023 2023-12-10T19:11:13.706 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:11:13.706 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:11:13.706 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: 8d5c864a-1785-4fbb-8bed-7f9006bb0d2b 2023-12-10T19:11:13.706 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: true 2023-12-10T19:11:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n ] 2023-12-10T19:11:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stress_write_image cluster2 mirror image_50 2023-12-10T19:11:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:11:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:11:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_50 2023-12-10T19:11:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-10T19:11:13.711 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local duration=5 2023-12-10T19:11:13.711 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T19:11:13.711 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_50 --debug-rbd=20 --debug-journaler=20 2023-12-10T19:11:13.960 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:100 IOs, 51387392 bytes 2023-12-10T19:11:14.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:200 IOs, 83505152 bytes 2023-12-10T19:11:14.325 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:300 IOs, 130157568 bytes 2023-12-10T19:11:14.498 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:400 IOs, 153962496 bytes 2023-12-10T19:11:14.690 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:500 IOs, 192013312 bytes 2023-12-10T19:11:14.938 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:600 IOs, 259948544 bytes 2023-12-10T19:11:15.133 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:700 IOs, 298231808 bytes 2023-12-10T19:11:15.351 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:800 IOs, 353120256 bytes 2023-12-10T19:11:15.633 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:900 IOs, 441140224 bytes 2023-12-10T19:11:15.794 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1000 IOs, 463491072 bytes 2023-12-10T19:11:15.953 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1100 IOs, 486583296 bytes 2023-12-10T19:11:16.066 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1200 IOs, 506528768 bytes 2023-12-10T19:11:16.247 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1300 IOs, 527065088 bytes 2023-12-10T19:11:16.422 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1400 IOs, 553738240 bytes 2023-12-10T19:11:16.592 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1500 IOs, 600095744 bytes 2023-12-10T19:11:16.807 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1600 IOs, 627953664 bytes 2023-12-10T19:11:17.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1700 IOs, 691115008 bytes 2023-12-10T19:11:17.240 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1800 IOs, 732506112 bytes 2023-12-10T19:11:17.372 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1900 IOs, 760804352 bytes 2023-12-10T19:11:17.687 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2000 IOs, 841505792 bytes 2023-12-10T19:11:17.875 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2100 IOs, 882035712 bytes 2023-12-10T19:11:18.058 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2200 IOs, 911388672 bytes 2023-12-10T19:11:18.257 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2300 IOs, 953947136 bytes 2023-12-10T19:11:18.367 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2400 IOs, 955112448 bytes 2023-12-10T19:11:18.526 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2500 IOs, 974713856 bytes 2023-12-10T19:11:18.716 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=124 2023-12-10T19:11:18.717 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T19:11:18.717 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 124 -eq 124 ] 2023-12-10T19:11:18.717 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:11:18.717 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_pool_images cluster2 mirror 50 2023-12-10T19:11:18.717 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:11:18.717 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:11:18.717 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_count=50 2023-12-10T19:11:18.717 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:11:18.717 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local count 2023-12-10T19:11:18.717 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local last_count=0 2023-12-10T19:11:18.717 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:11:18.717 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 1 40 2023-12-10T19:11:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 1 -ne 1 2023-12-10T19:11:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror mirror pool status 2023-12-10T19:11:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cut -d -f 2 2023-12-10T19:11:18.720 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep images: 2023-12-10T19:11:18.760 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: failed to query services: (13) Permission denied 2023-12-10T19:11:18.764 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ count=50 2023-12-10T19:11:18.764 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 50 = 50 2023-12-10T19:11:18.764 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:11:18.764 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_pool_healthy cluster2 mirror 2023-12-10T19:11:18.764 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:11:18.764 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:11:18.764 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:11:18.765 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state 2023-12-10T19:11:18.765 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 1 40 2023-12-10T19:11:18.765 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 1 -ne 1 2023-12-10T19:11:18.765 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror mirror pool status 2023-12-10T19:11:18.765 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep image health: 2023-12-10T19:11:18.765 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cut -d -f 3 2023-12-10T19:11:18.801 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: failed to query services: (13) Permission denied 2023-12-10T19:11:18.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ state=WARNING 2023-12-10T19:11:18.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test WARNING = ERROR 2023-12-10T19:11:18.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test WARNING = OK 2023-12-10T19:11:18.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 2 -ne 1 2023-12-10T19:11:18.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 30 2023-12-10T19:11:48.807 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror mirror pool status 2023-12-10T19:11:48.807 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep image health: 2023-12-10T19:11:48.807 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cut -d -f 3 2023-12-10T19:11:48.856 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: failed to query services: (13) Permission denied 2023-12-10T19:11:48.861 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ state=OK 2023-12-10T19:11:48.861 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test OK = ERROR 2023-12-10T19:11:48.861 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test OK = OK 2023-12-10T19:11:48.861 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:11:48.861 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_pool_images cluster1 mirror 50 2023-12-10T19:11:48.861 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:11:48.861 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:11:48.861 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_count=50 2023-12-10T19:11:48.861 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:11:48.862 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local count 2023-12-10T19:11:48.862 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local last_count=0 2023-12-10T19:11:48.862 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:11:48.862 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 1 40 2023-12-10T19:11:48.862 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 1 -ne 1 2023-12-10T19:11:48.863 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror mirror pool status 2023-12-10T19:11:48.863 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep images: 2023-12-10T19:11:48.863 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cut -d -f 2 2023-12-10T19:11:48.923 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: failed to query services: (13) Permission denied 2023-12-10T19:11:48.927 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ count=50 2023-12-10T19:11:48.927 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 50 = 50 2023-12-10T19:11:48.927 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:11:48.927 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_pool_healthy cluster1 mirror 2023-12-10T19:11:48.928 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:11:48.928 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:11:48.928 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:11:48.928 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state 2023-12-10T19:11:48.928 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 1 40 2023-12-10T19:11:48.930 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 1 -ne 1 2023-12-10T19:11:48.930 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror mirror pool status 2023-12-10T19:11:48.930 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep image health: 2023-12-10T19:11:48.931 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cut -d -f 3 2023-12-10T19:11:48.983 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: failed to query services: (13) Permission denied 2023-12-10T19:11:48.985 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ state=OK 2023-12-10T19:11:48.985 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test OK = ERROR 2023-12-10T19:11:48.985 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test OK = OK 2023-12-10T19:11:48.985 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:11:48.985 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ testlog TEST: compare many images 2023-12-10T19:11:48.985 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ tee -a /tmp/tmp.qNjRyMq75D/rbd-mirror.test.log 2023-12-10T19:11:48.985 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ date +%F %T 2023-12-10T19:11:48.985 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo 2023-12-10 19:11:48 TEST: compare many images 2023-12-10T19:11:48.986 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:2023-12-10 19:11:48 TEST: compare many images 2023-12-10T19:11:48.986 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 1 50 2023-12-10T19:11:48.986 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_1 2023-12-10T19:11:48.986 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror image_1 snap 2023-12-10T19:11:48.986 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:11:48.987 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:11:48.987 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_1 2023-12-10T19:11:48.987 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:11:48.987 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create image_1@snap --debug-rbd=20 --debug-journaler=20 2023-12-10T19:11:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_started cluster1 mirror image_1 2023-12-10T19:11:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:11:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:11:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_1 2023-12-10T19:11:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_state cluster1 mirror image_1 started 2023-12-10T19:11:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:11:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:11:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_1 2023-12-10T19:11:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=started 2023-12-10T19:11:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:11:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:11:50.614 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_replay_state cluster1 mirror image_1 started 2023-12-10T19:11:50.614 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:11:50.614 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:11:50.614 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_1 2023-12-10T19:11:50.614 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=started 2023-12-10T19:11:50.614 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_result 2023-12-10T19:11:50.614 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=stopped 2023-12-10T19:11:50.615 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep -i state 2023-12-10T19:11:50.615 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_1 2023-12-10T19:11:50.615 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_instance=cluster1 2023-12-10T19:11:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:11:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:11:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance=cluster1 2023-12-10T19:11:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loop_instance 2023-12-10T19:11:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:11:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T19:11:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T19:11:50.618 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_1 2023-12-10T19:11:50.618 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T19:11:50.618 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:11:50.618 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:11:50.618 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T19:11:50.618 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:11:50.618 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:11:50.618 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:11:50.618 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:11:50.618 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:11:50.618 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:11:50.619 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:11:50.619 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:11:50.619 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:11:50.619 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:11:50.619 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T19:11:50.619 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T19:11:50.619 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:11:50.619 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:11:50.619 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T19:11:50.619 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:11:50.619 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:11:50.619 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:11:50.619 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:11:50.619 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:11:50.619 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:11:50.620 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:11:50.620 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:11:50.620 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:11:50.620 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:11:50.620 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T19:11:50.645 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:11:50.645 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:11:50.645 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:11:50.645 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_1 2023-12-10T19:11:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:11:50.774 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ status_result= "state": "Replaying" 2023-12-10T19:11:50.774 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo "state": "Replaying" 2023-12-10T19:11:50.775 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep -i Replaying 2023-12-10T19:11:50.776 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "state": "Replaying" 2023-12-10T19:11:50.778 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=started 2023-12-10T19:11:50.778 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test started = started 2023-12-10T19:11:50.778 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:11:50.778 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_1 2023-12-10T19:11:50.778 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:11:50.778 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:11:50.778 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:11:50.778 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_1 2023-12-10T19:11:50.778 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = journal ] 2023-12-10T19:11:50.778 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = snapshot ] 2023-12-10T19:11:50.778 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_1 2023-12-10T19:11:50.778 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:11:50.778 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:11:50.778 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:11:50.779 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_1 2023-12-10T19:11:50.779 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster2-mirror-image_1.status 2023-12-10T19:11:50.779 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ + echo cluster2-mirror-image_1.status 2023-12-10T19:11:50.779 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:sed -e s|[/ ]|_|g 2023-12-10T19:11:50.780 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_1.status 2023-12-10T19:11:50.780 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-image_1.status 2023-12-10T19:11:50.780 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-image_1.status 2023-12-10T19:11:50.780 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:11:50.780 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_1.status 2023-12-10T19:11:50.780 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mirror_image_snapshot cluster2 mirror image_1 2023-12-10T19:11:50.781 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:11:50.781 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:11:50.781 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_1 2023-12-10T19:11:50.781 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_1 2023-12-10T19:11:51.653 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Snapshot ID: 119 2023-12-10T19:11:51.668 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_1 /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_1.status 2023-12-10T19:11:51.668 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:11:51.668 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:11:51.669 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_1 2023-12-10T19:11:51.669 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_1.status 2023-12-10T19:11:51.669 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_1 --format xml 2023-12-10T19:11:51.669 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:11:51.724 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-10T19:11:51.726 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snapshot_id=119 2023-12-10T19:11:51.726 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:11:51.726 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.2 2023-12-10T19:11:51.927 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_1.status 2023-12-10T19:11:51.927 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:11:51.927 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:11:51.928 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_1 2023-12-10T19:11:51.928 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_1.status 2023-12-10T19:11:51.928 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2023-12-10T19:11:51.928 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:11:51.996 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:11:51.999 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=68 2023-12-10T19:11:51.999 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 119 = 68 2023-12-10T19:11:52.000 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.4 2023-12-10T19:11:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_1.status 2023-12-10T19:11:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:11:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:11:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_1 2023-12-10T19:11:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_1.status 2023-12-10T19:11:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2023-12-10T19:11:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:11:52.468 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:11:52.470 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=68 2023-12-10T19:11:52.471 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 119 = 68 2023-12-10T19:11:52.471 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.8 2023-12-10T19:11:53.272 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_1.status 2023-12-10T19:11:53.272 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:11:53.272 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:11:53.273 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_1 2023-12-10T19:11:53.273 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_1.status 2023-12-10T19:11:53.273 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2023-12-10T19:11:53.273 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:11:53.339 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:11:53.339 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:11:53.343 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:11:53.343 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:11:53.343 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:11:53.343 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:11:53.343 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 119 = 2023-12-10T19:11:53.343 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1.6 2023-12-10T19:11:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_1.status 2023-12-10T19:11:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:11:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:11:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_1 2023-12-10T19:11:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_1.status 2023-12-10T19:11:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2023-12-10T19:11:54.946 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:11:55.012 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:11:55.016 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=119 2023-12-10T19:11:55.016 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 119 = 119 2023-12-10T19:11:55.016 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:11:55.016 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snap_present cluster1 mirror image_1 snap 2023-12-10T19:11:55.016 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:11:55.016 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:11:55.016 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_1 2023-12-10T19:11:55.016 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:11:55.016 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:11:55.016 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:11:56.017 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 info mirror/image_1@snap 2023-12-10T19:11:56.083 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_1': 2023-12-10T19:11:56.083 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:11:56.083 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:11:56.083 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 2 2023-12-10T19:11:56.083 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 10c06a220556 2023-12-10T19:11:56.083 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.10c06a220556 2023-12-10T19:11:56.083 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:11:56.083 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering, non-primary 2023-12-10T19:11:56.083 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:11:56.083 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:11:56.083 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:02:29 2023 2023-12-10T19:11:56.083 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:02:29 2023 2023-12-10T19:11:56.083 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:02:29 2023 2023-12-10T19:11:56.084 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: protected: False 2023-12-10T19:11:56.084 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:11:56.084 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:11:56.084 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: 8fd23019-e0bb-4243-ad49-4fbc841c8b88 2023-12-10T19:11:56.084 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: false 2023-12-10T19:11:56.088 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:11:56.088 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror image_1 snap 2023-12-10T19:11:56.088 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:11:56.088 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_1 2023-12-10T19:11:56.088 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:11:56.088 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:11:56.088 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_1.export 2023-12-10T19:11:56.088 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_1.export 2023-12-10T19:11:56.088 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_1.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_1.export 2023-12-10T19:11:56.089 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export image_1@snap /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_1.export 2023-12-10T19:11:56.299 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:11:56.310 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export image_1@snap /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_1.export 2023-12-10T19:11:56.622 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:11:56.636 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_1.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_1.export 2023-12-10T19:11:56.713 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_1.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_1.export 2023-12-10T19:11:56.748 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:11:56.748 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_2 2023-12-10T19:11:56.748 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror image_2 snap 2023-12-10T19:11:56.748 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:11:56.749 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:11:56.749 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_2 2023-12-10T19:11:56.749 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:11:56.749 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create image_2@snap --debug-rbd=20 --debug-journaler=20 2023-12-10T19:11:57.266 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_started cluster1 mirror image_2 2023-12-10T19:11:57.266 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:11:57.266 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:11:57.266 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_2 2023-12-10T19:11:57.266 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_state cluster1 mirror image_2 started 2023-12-10T19:11:57.266 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:11:57.266 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:11:57.266 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_2 2023-12-10T19:11:57.266 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=started 2023-12-10T19:11:57.266 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:11:57.266 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:11:58.267 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_replay_state cluster1 mirror image_2 started 2023-12-10T19:11:58.267 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:11:58.267 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:11:58.267 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_2 2023-12-10T19:11:58.267 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=started 2023-12-10T19:11:58.267 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_result 2023-12-10T19:11:58.268 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=stopped 2023-12-10T19:11:58.269 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep -i state 2023-12-10T19:11:58.269 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_2 2023-12-10T19:11:58.269 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_instance=cluster1 2023-12-10T19:11:58.269 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:11:58.269 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:11:58.269 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance=cluster1 2023-12-10T19:11:58.270 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loop_instance 2023-12-10T19:11:58.270 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:11:58.271 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T19:11:58.271 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T19:11:58.275 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_2 2023-12-10T19:11:58.276 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T19:11:58.276 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:11:58.276 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:11:58.276 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T19:11:58.276 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:11:58.276 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:11:58.277 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:11:58.277 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:11:58.277 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:11:58.277 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:11:58.277 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:11:58.277 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:11:58.277 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:11:58.278 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:11:58.278 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T19:11:58.278 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T19:11:58.278 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:11:58.278 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:11:58.278 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T19:11:58.279 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:11:58.279 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:11:58.279 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:11:58.279 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:11:58.279 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:11:58.279 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:11:58.279 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:11:58.279 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:11:58.279 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:11:58.280 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:11:58.280 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T19:11:58.296 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:11:58.296 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:11:58.296 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:11:58.297 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_2 2023-12-10T19:11:58.427 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:11:58.428 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ status_result= "state": "Replaying" 2023-12-10T19:11:58.428 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo "state": "Replaying" 2023-12-10T19:11:58.428 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep -i Replaying 2023-12-10T19:11:58.429 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "state": "Replaying" 2023-12-10T19:11:58.431 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=started 2023-12-10T19:11:58.431 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test started = started 2023-12-10T19:11:58.431 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:11:58.431 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_2 2023-12-10T19:11:58.431 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:11:58.431 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:11:58.431 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:11:58.431 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_2 2023-12-10T19:11:58.431 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = journal ] 2023-12-10T19:11:58.431 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = snapshot ] 2023-12-10T19:11:58.431 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_2 2023-12-10T19:11:58.432 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:11:58.432 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:11:58.432 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:11:58.432 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_2 2023-12-10T19:11:58.432 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster2-mirror-image_2.status 2023-12-10T19:11:58.432 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2-mirror-image_2.status 2023-12-10T19:11:58.432 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:11:58.433 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_2.status 2023-12-10T19:11:58.433 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-image_2.status 2023-12-10T19:11:58.433 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-image_2.status 2023-12-10T19:11:58.433 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:11:58.434 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_2.status 2023-12-10T19:11:58.434 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mirror_image_snapshot cluster2 mirror image_2 2023-12-10T19:11:58.434 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:11:58.434 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:11:58.434 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_2 2023-12-10T19:11:58.434 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_2 2023-12-10T19:11:59.326 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Snapshot ID: 121 2023-12-10T19:11:59.340 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_2 /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_2.status 2023-12-10T19:11:59.340 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:11:59.340 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:11:59.340 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_2 2023-12-10T19:11:59.340 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_2.status 2023-12-10T19:11:59.340 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_2 --format xml 2023-12-10T19:11:59.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:11:59.395 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-10T19:11:59.398 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snapshot_id=121 2023-12-10T19:11:59.398 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:11:59.398 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.2 2023-12-10T19:11:59.599 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_2.status 2023-12-10T19:11:59.599 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:11:59.599 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:11:59.599 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_2 2023-12-10T19:11:59.599 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_2.status 2023-12-10T19:11:59.599 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2023-12-10T19:11:59.599 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:11:59.668 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:11:59.668 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:11:59.670 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:11:59.670 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:11:59.671 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:11:59.671 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:11:59.671 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 121 = 2023-12-10T19:11:59.671 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.4 2023-12-10T19:12:00.072 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_2.status 2023-12-10T19:12:00.072 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:12:00.072 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:12:00.072 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_2 2023-12-10T19:12:00.072 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_2.status 2023-12-10T19:12:00.072 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2023-12-10T19:12:00.072 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:12:00.141 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:12:00.142 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:12:00.143 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:12:00.144 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:12:00.144 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:12:00.144 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:12:00.144 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 121 = 2023-12-10T19:12:00.144 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.8 2023-12-10T19:12:00.945 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_2.status 2023-12-10T19:12:00.945 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:12:00.945 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:12:00.946 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_2 2023-12-10T19:12:00.946 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_2.status 2023-12-10T19:12:00.946 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2023-12-10T19:12:00.946 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:12:01.012 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:12:01.012 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:12:01.014 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:12:01.014 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:12:01.014 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:12:01.015 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:12:01.015 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 121 = 2023-12-10T19:12:01.015 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1.6 2023-12-10T19:12:02.616 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_2.status 2023-12-10T19:12:02.616 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:12:02.616 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:12:02.616 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_2 2023-12-10T19:12:02.616 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_2.status 2023-12-10T19:12:02.617 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2023-12-10T19:12:02.617 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:12:02.684 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:12:02.688 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=121 2023-12-10T19:12:02.688 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 121 = 121 2023-12-10T19:12:02.689 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:12:02.689 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snap_present cluster1 mirror image_2 snap 2023-12-10T19:12:02.689 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:12:02.689 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:12:02.689 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_2 2023-12-10T19:12:02.689 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:12:02.689 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:12:02.689 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:12:03.688 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 info mirror/image_2@snap 2023-12-10T19:12:03.956 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_2': 2023-12-10T19:12:03.956 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:12:03.956 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:12:03.956 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 2 2023-12-10T19:12:03.956 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 10e86b2e278e 2023-12-10T19:12:03.956 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.10e86b2e278e 2023-12-10T19:12:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:12:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering, non-primary 2023-12-10T19:12:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:12:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:12:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:02:43 2023 2023-12-10T19:12:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:02:43 2023 2023-12-10T19:12:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:02:43 2023 2023-12-10T19:12:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: protected: False 2023-12-10T19:12:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:12:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:12:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: 13cd2888-c5a1-4f79-8bc5-a118e5cace49 2023-12-10T19:12:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: false 2023-12-10T19:12:03.962 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:12:03.962 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror image_2 snap 2023-12-10T19:12:03.962 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:12:03.962 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_2 2023-12-10T19:12:03.962 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:12:03.962 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:12:03.962 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_2.export 2023-12-10T19:12:03.962 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_2.export 2023-12-10T19:12:03.962 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_2.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_2.export 2023-12-10T19:12:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export image_2@snap /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_2.export 2023-12-10T19:12:04.169 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:12:04.178 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export image_2@snap /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_2.export 2023-12-10T19:12:04.536 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:12:04.542 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_2.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_2.export 2023-12-10T19:12:04.617 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_2.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_2.export 2023-12-10T19:12:04.656 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:12:04.656 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_3 2023-12-10T19:12:04.657 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror image_3 snap 2023-12-10T19:12:04.657 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:12:04.657 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:12:04.657 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_3 2023-12-10T19:12:04.657 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:12:04.657 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create image_3@snap --debug-rbd=20 --debug-journaler=20 2023-12-10T19:12:05.301 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_started cluster1 mirror image_3 2023-12-10T19:12:05.301 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:12:05.301 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:12:05.301 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_3 2023-12-10T19:12:05.301 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_state cluster1 mirror image_3 started 2023-12-10T19:12:05.301 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:12:05.301 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:12:05.302 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_3 2023-12-10T19:12:05.302 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=started 2023-12-10T19:12:05.302 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:12:05.302 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:12:06.302 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_replay_state cluster1 mirror image_3 started 2023-12-10T19:12:06.302 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:12:06.302 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:12:06.302 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_3 2023-12-10T19:12:06.302 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=started 2023-12-10T19:12:06.302 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_result 2023-12-10T19:12:06.302 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=stopped 2023-12-10T19:12:06.303 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ + admin_daemons cluster1 rbd mirror status mirror/image_3 2023-12-10T19:12:06.303 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:grep -i state 2023-12-10T19:12:06.303 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_instance=cluster1 2023-12-10T19:12:06.303 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:12:06.303 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:12:06.303 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance=cluster1 2023-12-10T19:12:06.303 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loop_instance 2023-12-10T19:12:06.303 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:12:06.303 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T19:12:06.305 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T19:12:06.305 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_3 2023-12-10T19:12:06.305 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T19:12:06.305 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:12:06.305 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:12:06.305 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T19:12:06.305 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:12:06.305 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:12:06.305 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:12:06.305 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:12:06.305 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:12:06.305 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:12:06.305 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:12:06.306 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:12:06.306 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:12:06.306 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:12:06.306 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T19:12:06.306 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T19:12:06.306 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:12:06.306 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:12:06.306 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T19:12:06.307 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:12:06.307 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:12:06.307 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:12:06.307 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:12:06.307 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:12:06.307 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:12:06.307 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:12:06.307 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:12:06.307 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:12:06.307 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:12:06.307 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T19:12:06.326 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:12:06.326 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:12:06.327 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:12:06.327 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_3 2023-12-10T19:12:06.437 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:no valid command found; 3 closest matches: 2023-12-10T19:12:06.438 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status mirror/image_34 2023-12-10T19:12:06.438 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status mirror/image_35 2023-12-10T19:12:06.438 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status mirror/image_36 2023-12-10T19:12:06.438 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:12:06.440 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_3 2023-12-10T19:12:06.440 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:1 2023-12-10T19:12:06.440 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:12:06.440 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:12:06.440 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-10T19:12:06.440 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:12:06.440 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:12:06.440 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:12:06.440 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:12:06.440 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:12:06.441 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:12:06.441 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:12:06.441 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:12:06.441 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:12:06.441 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:12:06.441 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-10T19:12:06.441 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:1 2023-12-10T19:12:06.441 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:12:06.441 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:12:06.441 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-10T19:12:06.441 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:12:06.441 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:12:06.441 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:12:06.441 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:12:06.441 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:12:06.441 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:12:06.442 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:12:06.442 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:12:06.442 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:12:06.442 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:12:06.442 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-10T19:12:06.463 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:12:06.463 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:12:06.464 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:12:06.464 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_3 2023-12-10T19:12:06.585 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:no valid command found; 3 closest matches: 2023-12-10T19:12:06.586 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status mirror/image_31 2023-12-10T19:12:06.586 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status mirror/image_32 2023-12-10T19:12:06.586 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status mirror/image_39 2023-12-10T19:12:06.586 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:12:06.588 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_3 2023-12-10T19:12:06.588 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:2 2023-12-10T19:12:06.588 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:12:06.588 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:12:06.588 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-10T19:12:06.588 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:12:06.588 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:12:06.589 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:12:06.589 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:12:06.589 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:12:06.589 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:12:06.589 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:12:06.589 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:12:06.589 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:12:06.589 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:12:06.589 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-10T19:12:06.589 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:2 2023-12-10T19:12:06.589 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:12:06.589 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:12:06.589 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-10T19:12:06.589 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:12:06.590 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:12:06.590 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:12:06.590 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:12:06.590 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:12:06.590 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:12:06.590 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:12:06.590 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:12:06.590 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:12:06.590 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:12:06.590 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-10T19:12:06.613 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:12:06.613 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:12:06.613 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:12:06.613 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_3 2023-12-10T19:12:06.740 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:12:06.741 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ status_result= "state": "Replaying" 2023-12-10T19:12:06.741 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo "state": "Replaying" 2023-12-10T19:12:06.741 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep -i Replaying 2023-12-10T19:12:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "state": "Replaying" 2023-12-10T19:12:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=started 2023-12-10T19:12:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test started = started 2023-12-10T19:12:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:12:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_3 2023-12-10T19:12:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:12:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:12:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:12:06.744 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_3 2023-12-10T19:12:06.744 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = journal ] 2023-12-10T19:12:06.744 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = snapshot ] 2023-12-10T19:12:06.744 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_3 2023-12-10T19:12:06.744 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:12:06.744 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:12:06.744 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:12:06.744 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_3 2023-12-10T19:12:06.744 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster2-mirror-image_3.status 2023-12-10T19:12:06.744 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2-mirror-image_3.status 2023-12-10T19:12:06.744 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:12:06.745 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_3.status 2023-12-10T19:12:06.745 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-image_3.status 2023-12-10T19:12:06.745 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-image_3.status 2023-12-10T19:12:06.745 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:12:06.746 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_3.status 2023-12-10T19:12:06.746 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mirror_image_snapshot cluster2 mirror image_3 2023-12-10T19:12:06.746 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:12:06.746 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:12:06.746 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_3 2023-12-10T19:12:06.747 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_3 2023-12-10T19:12:07.337 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Snapshot ID: 123 2023-12-10T19:12:07.351 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_3 /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_3.status 2023-12-10T19:12:07.352 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:12:07.352 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:12:07.352 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_3 2023-12-10T19:12:07.352 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_3.status 2023-12-10T19:12:07.353 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_3 --format xml 2023-12-10T19:12:07.353 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:12:07.410 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-10T19:12:07.412 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snapshot_id=123 2023-12-10T19:12:07.412 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:12:07.412 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.2 2023-12-10T19:12:07.613 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_3.status 2023-12-10T19:12:07.613 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:12:07.613 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:12:07.613 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_3 2023-12-10T19:12:07.613 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_3.status 2023-12-10T19:12:07.614 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2023-12-10T19:12:07.614 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:12:07.683 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:12:07.683 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:12:07.685 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:12:07.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:12:07.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:12:07.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:12:07.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 123 = 2023-12-10T19:12:07.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.4 2023-12-10T19:12:08.087 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_3.status 2023-12-10T19:12:08.087 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:12:08.087 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:12:08.087 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_3 2023-12-10T19:12:08.087 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_3.status 2023-12-10T19:12:08.088 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2023-12-10T19:12:08.088 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:12:08.365 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:12:08.366 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:12:08.368 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:12:08.368 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:12:08.368 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:12:08.368 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:12:08.369 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 123 = 2023-12-10T19:12:08.369 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.8 2023-12-10T19:12:09.170 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_3.status 2023-12-10T19:12:09.170 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:12:09.170 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:12:09.170 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_3 2023-12-10T19:12:09.170 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_3.status 2023-12-10T19:12:09.171 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2023-12-10T19:12:09.171 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:12:09.240 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:12:09.243 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=123 2023-12-10T19:12:09.243 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 123 = 123 2023-12-10T19:12:09.243 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:12:09.243 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snap_present cluster1 mirror image_3 snap 2023-12-10T19:12:09.243 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:12:09.243 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:12:09.243 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_3 2023-12-10T19:12:09.243 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:12:09.243 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:12:09.243 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:12:10.244 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 info mirror/image_3@snap 2023-12-10T19:12:10.310 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_3': 2023-12-10T19:12:10.310 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:12:10.310 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:12:10.310 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 2 2023-12-10T19:12:10.310 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 117e98a889d 2023-12-10T19:12:10.310 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.117e98a889d 2023-12-10T19:12:10.310 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:12:10.310 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering, non-primary 2023-12-10T19:12:10.310 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:12:10.310 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:12:10.310 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:02:53 2023 2023-12-10T19:12:10.310 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:02:53 2023 2023-12-10T19:12:10.311 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:02:53 2023 2023-12-10T19:12:10.311 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: protected: False 2023-12-10T19:12:10.311 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:12:10.311 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:12:10.311 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: 3140c852-fad6-4a00-a40c-46869faa3ea1 2023-12-10T19:12:10.311 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: false 2023-12-10T19:12:10.314 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:12:10.314 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror image_3 snap 2023-12-10T19:12:10.314 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:12:10.314 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_3 2023-12-10T19:12:10.314 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:12:10.314 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:12:10.314 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_3.export 2023-12-10T19:12:10.314 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_3.export 2023-12-10T19:12:10.314 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_3.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_3.export 2023-12-10T19:12:10.315 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export image_3@snap /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_3.export 2023-12-10T19:12:10.526 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:12:10.534 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export image_3@snap /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_3.export 2023-12-10T19:12:10.797 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:12:10.813 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_3.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_3.export 2023-12-10T19:12:10.888 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_3.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_3.export 2023-12-10T19:12:10.929 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:12:10.929 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_4 2023-12-10T19:12:10.929 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror image_4 snap 2023-12-10T19:12:10.929 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:12:10.929 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:12:10.929 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_4 2023-12-10T19:12:10.929 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:12:10.930 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create image_4@snap --debug-rbd=20 --debug-journaler=20 2023-12-10T19:12:11.625 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_started cluster1 mirror image_4 2023-12-10T19:12:11.625 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:12:11.625 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:12:11.625 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_4 2023-12-10T19:12:11.626 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_state cluster1 mirror image_4 started 2023-12-10T19:12:11.626 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:12:11.626 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:12:11.626 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_4 2023-12-10T19:12:11.626 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=started 2023-12-10T19:12:11.626 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:12:11.626 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:12:12.627 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_replay_state cluster1 mirror image_4 started 2023-12-10T19:12:12.627 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:12:12.627 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:12:12.627 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_4 2023-12-10T19:12:12.627 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=started 2023-12-10T19:12:12.627 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_result 2023-12-10T19:12:12.627 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=stopped 2023-12-10T19:12:12.628 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_4 2023-12-10T19:12:12.628 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_instance=cluster1 2023-12-10T19:12:12.628 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:12:12.628 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:12:12.628 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ + local instance=cluster1 2023-12-10T19:12:12.628 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loop_instance 2023-12-10T19:12:12.628 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:grep+ -i state 2023-12-10T19:12:12.628 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:sleep 0 2023-12-10T19:12:12.629 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T19:12:12.629 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T19:12:12.631 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_4 2023-12-10T19:12:12.631 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T19:12:12.631 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:12:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:12:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T19:12:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:12:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:12:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:12:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:12:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:12:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:12:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:12:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:12:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:12:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:12:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T19:12:12.633 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T19:12:12.633 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:12:12.633 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:12:12.633 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T19:12:12.633 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:12:12.633 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:12:12.633 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:12:12.633 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:12:12.633 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:12:12.633 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:12:12.633 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:12:12.633 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:12:12.633 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:12:12.633 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:12:12.633 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T19:12:12.654 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:12:12.654 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:12:12.655 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:12:12.655 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_4 2023-12-10T19:12:12.788 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:12:12.789 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ status_result= "state": "Replaying" 2023-12-10T19:12:12.789 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo "state": "Replaying" 2023-12-10T19:12:12.789 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep -i Replaying 2023-12-10T19:12:12.790 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "state": "Replaying" 2023-12-10T19:12:12.791 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=started 2023-12-10T19:12:12.791 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test started = started 2023-12-10T19:12:12.792 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:12:12.792 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_4 2023-12-10T19:12:12.792 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:12:12.792 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:12:12.792 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:12:12.792 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_4 2023-12-10T19:12:12.792 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = journal ] 2023-12-10T19:12:12.792 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = snapshot ] 2023-12-10T19:12:12.792 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_4 2023-12-10T19:12:12.792 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:12:12.792 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:12:12.792 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:12:12.792 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_4 2023-12-10T19:12:12.792 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster2-mirror-image_4.status 2023-12-10T19:12:12.792 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2-mirror-image_4.status 2023-12-10T19:12:12.793 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:12:12.794 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_4.status 2023-12-10T19:12:12.794 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-image_4.status 2023-12-10T19:12:12.794 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-image_4.status 2023-12-10T19:12:12.794 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:12:12.795 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_4.status 2023-12-10T19:12:12.795 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mirror_image_snapshot cluster2 mirror image_4 2023-12-10T19:12:12.795 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:12:12.795 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:12:12.795 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_4 2023-12-10T19:12:12.795 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_4 2023-12-10T19:12:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Snapshot ID: 125 2023-12-10T19:12:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_4 /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_4.status 2023-12-10T19:12:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:12:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:12:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_4 2023-12-10T19:12:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_4.status 2023-12-10T19:12:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_4 --format xml 2023-12-10T19:12:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:12:13.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-10T19:12:13.734 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snapshot_id=125 2023-12-10T19:12:13.734 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:12:13.735 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.2 2023-12-10T19:12:13.936 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_4.status 2023-12-10T19:12:13.936 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:12:13.936 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:12:13.936 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_4 2023-12-10T19:12:13.936 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_4.status 2023-12-10T19:12:13.936 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2023-12-10T19:12:13.936 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:12:14.008 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:12:14.011 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=71 2023-12-10T19:12:14.012 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 125 = 71 2023-12-10T19:12:14.012 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.4 2023-12-10T19:12:14.413 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_4.status 2023-12-10T19:12:14.414 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:12:14.414 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:12:14.414 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_4 2023-12-10T19:12:14.414 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_4.status 2023-12-10T19:12:14.414 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2023-12-10T19:12:14.414 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:12:14.490 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:12:14.490 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:12:14.492 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:12:14.492 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:12:14.493 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:12:14.493 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:12:14.493 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 125 = 2023-12-10T19:12:14.493 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.8 2023-12-10T19:12:15.294 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_4.status 2023-12-10T19:12:15.294 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:12:15.294 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:12:15.294 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_4 2023-12-10T19:12:15.294 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_4.status 2023-12-10T19:12:15.294 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2023-12-10T19:12:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:12:15.362 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:12:15.363 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:12:15.365 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:12:15.365 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:12:15.365 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:12:15.365 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:12:15.365 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 125 = 2023-12-10T19:12:15.365 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1.6 2023-12-10T19:12:16.966 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_4.status 2023-12-10T19:12:16.967 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:12:16.967 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:12:16.967 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_4 2023-12-10T19:12:16.967 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_4.status 2023-12-10T19:12:16.967 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2023-12-10T19:12:16.967 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:12:17.033 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:12:17.035 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=125 2023-12-10T19:12:17.035 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 125 = 125 2023-12-10T19:12:17.036 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:12:17.036 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snap_present cluster1 mirror image_4 snap 2023-12-10T19:12:17.036 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:12:17.036 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:12:17.036 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_4 2023-12-10T19:12:17.036 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:12:17.036 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:12:17.036 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:12:18.036 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 info mirror/image_4@snap 2023-12-10T19:12:18.110 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_4': 2023-12-10T19:12:18.110 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:12:18.110 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:12:18.110 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 2 2023-12-10T19:12:18.110 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 1200201bf36a 2023-12-10T19:12:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.1200201bf36a 2023-12-10T19:12:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:12:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering, non-primary 2023-12-10T19:12:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:12:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:12:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:03:06 2023 2023-12-10T19:12:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:03:06 2023 2023-12-10T19:12:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:03:06 2023 2023-12-10T19:12:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: protected: False 2023-12-10T19:12:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:12:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:12:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: 2e46723e-74b1-4e29-b6e7-8617064dd725 2023-12-10T19:12:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: false 2023-12-10T19:12:18.115 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:12:18.115 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror image_4 snap 2023-12-10T19:12:18.115 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:12:18.115 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_4 2023-12-10T19:12:18.115 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:12:18.115 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:12:18.115 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_4.export 2023-12-10T19:12:18.115 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_4.export 2023-12-10T19:12:18.115 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_4.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_4.export 2023-12-10T19:12:18.115 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export image_4@snap /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_4.export 2023-12-10T19:12:18.320 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:12:18.329 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export image_4@snap /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_4.export 2023-12-10T19:12:18.638 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:12:18.653 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_4.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_4.export 2023-12-10T19:12:18.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_4.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_4.export 2023-12-10T19:12:18.769 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:12:18.769 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_5 2023-12-10T19:12:18.770 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror image_5 snap 2023-12-10T19:12:18.770 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:12:18.770 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:12:18.770 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_5 2023-12-10T19:12:18.770 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:12:18.770 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create image_5@snap --debug-rbd=20 --debug-journaler=20 2023-12-10T19:12:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_started cluster1 mirror image_5 2023-12-10T19:12:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:12:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:12:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_5 2023-12-10T19:12:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_state cluster1 mirror image_5 started 2023-12-10T19:12:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:12:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:12:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_5 2023-12-10T19:12:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=started 2023-12-10T19:12:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:12:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:12:20.669 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_replay_state cluster1 mirror image_5 started 2023-12-10T19:12:20.670 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:12:20.670 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:12:20.670 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_5 2023-12-10T19:12:20.670 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=started 2023-12-10T19:12:20.670 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_result 2023-12-10T19:12:20.670 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=stopped 2023-12-10T19:12:20.671 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_5 2023-12-10T19:12:20.671 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_instance=cluster1 2023-12-10T19:12:20.671 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:12:20.671 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:12:20.671 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance=cluster1 2023-12-10T19:12:20.671 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loop_instance 2023-12-10T19:12:20.671 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:12:20.671 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep -i state 2023-12-10T19:12:20.673 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T19:12:20.673 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T19:12:20.673 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_5 2023-12-10T19:12:20.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T19:12:20.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:12:20.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:12:20.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T19:12:20.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:12:20.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:12:20.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:12:20.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:12:20.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:12:20.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:12:20.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:12:20.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:12:20.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:12:20.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:12:20.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T19:12:20.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T19:12:20.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:12:20.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:12:20.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T19:12:20.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:12:20.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:12:20.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:12:20.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:12:20.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:12:20.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:12:20.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:12:20.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:12:20.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:12:20.676 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:12:20.676 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T19:12:20.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:12:20.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:12:20.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:12:20.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_5 2023-12-10T19:12:20.825 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:12:20.826 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ status_result= "state": "Replaying" 2023-12-10T19:12:20.826 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo "state": "Replaying" 2023-12-10T19:12:20.826 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep -i Replaying 2023-12-10T19:12:20.826 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "state": "Replaying" 2023-12-10T19:12:20.828 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=started 2023-12-10T19:12:20.828 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test started = started 2023-12-10T19:12:20.828 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:12:20.828 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_5 2023-12-10T19:12:20.828 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:12:20.828 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:12:20.828 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:12:20.828 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_5 2023-12-10T19:12:20.828 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = journal ] 2023-12-10T19:12:20.828 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = snapshot ] 2023-12-10T19:12:20.828 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_5 2023-12-10T19:12:20.828 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:12:20.828 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:12:20.828 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:12:20.829 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_5 2023-12-10T19:12:20.829 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster2-mirror-image_5.status 2023-12-10T19:12:20.829 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2-mirror-image_5.status 2023-12-10T19:12:20.829 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:12:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_5.status 2023-12-10T19:12:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-image_5.status 2023-12-10T19:12:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-image_5.status 2023-12-10T19:12:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:12:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_5.status 2023-12-10T19:12:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mirror_image_snapshot cluster2 mirror image_5 2023-12-10T19:12:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:12:20.831 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:12:20.831 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_5 2023-12-10T19:12:20.831 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_5 2023-12-10T19:12:21.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Snapshot ID: 127 2023-12-10T19:12:21.712 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_5 /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_5.status 2023-12-10T19:12:21.712 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:12:21.712 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:12:21.713 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_5 2023-12-10T19:12:21.713 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_5.status 2023-12-10T19:12:21.713 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_5 --format xml 2023-12-10T19:12:21.713 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:12:21.770 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-10T19:12:21.773 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snapshot_id=127 2023-12-10T19:12:21.773 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:12:21.773 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.2 2023-12-10T19:12:21.860 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-12-10T19:12:21.860 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-12-10T19:12:21.860 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2023-12-10T19:12:21.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr:2023-12-10T19:12:21.864+0000 7f4590c07640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 51520) UID: 1000 2023-12-10T19:12:21.974 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_5.status 2023-12-10T19:12:21.975 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:12:21.975 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:12:21.975 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_5 2023-12-10T19:12:21.975 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_5.status 2023-12-10T19:12:21.975 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2023-12-10T19:12:21.975 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:12:22.041 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:12:22.042 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=72 2023-12-10T19:12:22.043 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 127 = 72 2023-12-10T19:12:22.043 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.4 2023-12-10T19:12:22.444 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_5.status 2023-12-10T19:12:22.444 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:12:22.444 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:12:22.444 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_5 2023-12-10T19:12:22.444 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_5.status 2023-12-10T19:12:22.444 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2023-12-10T19:12:22.444 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:12:22.509 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:12:22.509 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:12:22.511 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:12:22.511 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:12:22.511 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:12:22.511 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:12:22.511 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 127 = 2023-12-10T19:12:22.511 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.8 2023-12-10T19:12:23.313 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_5.status 2023-12-10T19:12:23.313 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:12:23.314 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:12:23.314 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_5 2023-12-10T19:12:23.314 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_5.status 2023-12-10T19:12:23.314 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2023-12-10T19:12:23.314 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:12:23.385 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:12:23.385 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:12:23.387 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:12:23.387 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:12:23.388 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:12:23.388 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:12:23.388 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 127 = 2023-12-10T19:12:23.388 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1.6 2023-12-10T19:12:24.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_5.status 2023-12-10T19:12:24.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:12:24.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:12:24.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_5 2023-12-10T19:12:24.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_5.status 2023-12-10T19:12:24.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2023-12-10T19:12:24.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:12:25.057 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:12:25.057 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:12:25.059 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:12:25.059 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:12:25.059 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:12:25.060 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:12:25.060 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 127 = 2023-12-10T19:12:25.060 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 2 2023-12-10T19:12:26.862 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-12-10T19:12:26.862 INFO:teuthology.orchestra.run:waiting for 600 2023-12-10T19:12:26.862 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-12-10T19:12:26.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-10T19:12:26.863 DEBUG:teuthology.orchestra.run.smithi076:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-10T19:12:26.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-10T19:12:26.866 INFO:tasks.rbd_mirror_thrash:waiting for 88 secs before thrashing 2023-12-10T19:12:27.061 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_5.status 2023-12-10T19:12:27.061 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:12:27.061 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:12:27.061 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_5 2023-12-10T19:12:27.061 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_5.status 2023-12-10T19:12:27.061 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster+ cluster1 snap list --all mirror/image_5 --format xml 2023-12-10T19:12:27.061 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:12:27.135 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:12:27.135 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:12:27.137 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:12:27.137 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:12:27.137 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:12:27.137 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:12:27.137 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 127 = 2023-12-10T19:12:27.137 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 2 2023-12-10T19:12:29.139 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_5.status 2023-12-10T19:12:29.139 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:12:29.139 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:12:29.139 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_5 2023-12-10T19:12:29.139 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_5.status 2023-12-10T19:12:29.139 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2023-12-10T19:12:29.139 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:12:29.205 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:12:29.205 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:12:29.207 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:12:29.207 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:12:29.208 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:12:29.208 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:12:29.208 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 127 = 2023-12-10T19:12:29.208 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 4 2023-12-10T19:12:33.209 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_5.status 2023-12-10T19:12:33.209 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:12:33.210 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:12:33.210 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_5 2023-12-10T19:12:33.210 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_5.status 2023-12-10T19:12:33.210 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2023-12-10T19:12:33.210 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:12:33.276 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:12:33.276 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:12:33.278 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:12:33.278 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:12:33.278 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:12:33.279 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:12:33.279 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 127 = 2023-12-10T19:12:33.279 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 4 2023-12-10T19:12:37.280 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_5.status 2023-12-10T19:12:37.280 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:12:37.281 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:12:37.281 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_5 2023-12-10T19:12:37.281 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_5.status 2023-12-10T19:12:37.281 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2023-12-10T19:12:37.281 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:12:37.346 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:12:37.346 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:12:37.348 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:12:37.348 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:12:37.349 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:12:37.349 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:12:37.349 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 127 = 2023-12-10T19:12:37.349 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 8 2023-12-10T19:12:45.350 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_5.status 2023-12-10T19:12:45.350 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:12:45.350 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:12:45.351 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_5 2023-12-10T19:12:45.351 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_5.status 2023-12-10T19:12:45.351 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2023-12-10T19:12:45.351 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:12:45.421 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:12:45.421 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:12:45.424 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:12:45.425 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:12:45.425 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:12:45.425 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:12:45.425 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 127 = 2023-12-10T19:12:45.425 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 8 2023-12-10T19:12:52.857 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.8233856461943927) > weight (0.25) 2023-12-10T19:12:52.858 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2023-12-10T19:12:52.858 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2023-12-10T19:12:52.858 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2023-12-10T19:12:52.859 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr:2023-12-10T19:12:52.866+0000 7fe4da88a640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 48104) UID: 1000 2023-12-10T19:12:53.426 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_5.status 2023-12-10T19:12:53.426 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:12:53.426 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:12:53.426 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_5 2023-12-10T19:12:53.426 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_5.status 2023-12-10T19:12:53.426 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2023-12-10T19:12:53.426 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:12:53.496 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:12:53.496 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:12:53.499 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:12:53.500 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:12:53.500 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:12:53.500 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:12:53.500 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 127 = 2023-12-10T19:12:53.500 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 16 2023-12-10T19:12:55.663 INFO:tasks.ceph.cluster1.osd.1.smithi033.stderr:2023-12-10T19:12:55.662+0000 7f06a5518640 -1 reset not still connected to 0x563fbc3e9520 2023-12-10T19:12:55.663 INFO:tasks.ceph.cluster1.osd.1.smithi033.stderr:2023-12-10T19:12:55.662+0000 7f06a5518640 -1 reset not still connected to 0x563fbcc0e0d0 2023-12-10T19:12:55.663 INFO:tasks.ceph.cluster1.osd.1.smithi033.stderr:2023-12-10T19:12:55.662+0000 7f06a5518640 -1 reset not still connected to 0x563fbcc0e680 2023-12-10T19:12:55.663 INFO:tasks.ceph.cluster1.osd.1.smithi033.stderr:2023-12-10T19:12:55.662+0000 7f06a5518640 -1 reset not still connected to 0x563fbdd90a90 2023-12-10T19:12:55.663 INFO:tasks.ceph.cluster1.osd.1.smithi033.stderr:2023-12-10T19:12:55.662+0000 7f06a5518640 -1 reset not still connected to 0x563fc2b3d450 2023-12-10T19:12:55.664 INFO:tasks.ceph.cluster1.osd.1.smithi033.stderr:2023-12-10T19:12:55.662+0000 7f06a5518640 -1 reset not still connected to 0x563fc2c52680 2023-12-10T19:12:55.664 INFO:tasks.ceph.cluster1.osd.1.smithi033.stderr:2023-12-10T19:12:55.662+0000 7f06a5518640 -1 reset not still connected to 0x563fc2c53ee0 2023-12-10T19:12:58.859 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2023-12-10T19:12:58.859 INFO:teuthology.orchestra.run:waiting for 600 2023-12-10T19:12:58.859 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2023-12-10T19:12:58.860 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-12-10T19:12:58.860 DEBUG:teuthology.orchestra.run.smithi076:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-12-10T19:12:58.863 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-12-10T19:12:58.863 INFO:tasks.rbd_mirror_thrash:waiting for 88 secs before thrashing 2023-12-10T19:13:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_5.status 2023-12-10T19:13:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_5 2023-12-10T19:13:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_5.status 2023-12-10T19:13:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2023-12-10T19:13:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:13:09.570 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:13:09.572 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=127 2023-12-10T19:13:09.572 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 127 = 127 2023-12-10T19:13:09.572 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:13:09.572 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snap_present cluster1 mirror image_5 snap 2023-12-10T19:13:09.573 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:09.573 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:09.573 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_5 2023-12-10T19:13:09.573 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:13:09.573 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:13:09.573 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:13:10.573 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 info mirror/image_5@snap 2023-12-10T19:13:10.638 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_5': 2023-12-10T19:13:10.638 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:13:10.638 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:13:10.638 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 2 2023-12-10T19:13:10.638 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 10c0ad9428e6 2023-12-10T19:13:10.639 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.10c0ad9428e6 2023-12-10T19:13:10.639 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:13:10.639 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering, non-primary 2023-12-10T19:13:10.639 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:13:10.639 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:13:10.639 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:03:21 2023 2023-12-10T19:13:10.639 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:03:21 2023 2023-12-10T19:13:10.639 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:03:21 2023 2023-12-10T19:13:10.639 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: protected: False 2023-12-10T19:13:10.639 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:13:10.639 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:13:10.639 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: c911f848-6f73-4853-a1d8-243e89af66d6 2023-12-10T19:13:10.639 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: false 2023-12-10T19:13:10.642 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:13:10.643 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror image_5 snap 2023-12-10T19:13:10.643 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:10.643 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_5 2023-12-10T19:13:10.643 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:13:10.643 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:13:10.643 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_5.export 2023-12-10T19:13:10.643 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_5.export 2023-12-10T19:13:10.643 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_5.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_5.export 2023-12-10T19:13:10.643 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export image_5@snap /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_5.export 2023-12-10T19:13:10.829 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:13:10.837 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export image_5@snap /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_5.export 2023-12-10T19:13:11.136 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:13:11.152 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_5.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_5.export 2023-12-10T19:13:11.228 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_5.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_5.export 2023-12-10T19:13:11.267 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:13:11.268 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_6 2023-12-10T19:13:11.268 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror image_6 snap 2023-12-10T19:13:11.268 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:13:11.268 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:11.268 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_6 2023-12-10T19:13:11.268 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:13:11.268 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create image_6@snap --debug-rbd=20 --debug-journaler=20 2023-12-10T19:13:11.649 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_started cluster1 mirror image_6 2023-12-10T19:13:11.649 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:11.650 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:11.650 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_6 2023-12-10T19:13:11.650 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_state cluster1 mirror image_6 started 2023-12-10T19:13:11.650 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:11.650 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:11.650 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_6 2023-12-10T19:13:11.650 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=started 2023-12-10T19:13:11.650 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:13:11.650 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:13:12.651 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_replay_state cluster1 mirror image_6 started 2023-12-10T19:13:12.651 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:12.651 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:12.651 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_6 2023-12-10T19:13:12.651 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=started 2023-12-10T19:13:12.651 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_result 2023-12-10T19:13:12.651 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=stopped 2023-12-10T19:13:12.651 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemons cluster1 rbd+ mirror status mirror/image_6 2023-12-10T19:13:12.652 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_instance=cluster1 2023-12-10T19:13:12.652 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:grep+ -ishift state 2023-12-10T19:13:12.652 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:13:12.652 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:12.652 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance=cluster1 2023-12-10T19:13:12.652 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loop_instance 2023-12-10T19:13:12.652 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:13:12.653 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T19:13:12.654 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T19:13:12.654 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_6 2023-12-10T19:13:12.654 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T19:13:12.654 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:13:12.654 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:13:12.654 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T19:13:12.654 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:13:12.654 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:13:12.654 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:13:12.654 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:13:12.654 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:13:12.654 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:13:12.654 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:13:12.654 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:13:12.655 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:13:12.655 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:13:12.655 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T19:13:12.655 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T19:13:12.655 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:12.655 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:13:12.655 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T19:13:12.655 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:13:12.655 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:13:12.655 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:13:12.655 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:13:12.655 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:13:12.655 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:13:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:13:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:13:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:13:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:13:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T19:13:12.676 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:13:12.676 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:13:12.676 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:13:12.677 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_6 2023-12-10T19:13:12.789 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2023-12-10T19:13:12.789 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:13:12.789 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:13:12.792 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_6 2023-12-10T19:13:12.792 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:1 2023-12-10T19:13:12.792 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:13:12.792 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:13:12.792 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-10T19:13:12.792 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:13:12.792 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:13:12.792 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:13:12.792 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:13:12.792 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:13:12.793 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:13:12.793 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:13:12.793 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:13:12.793 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:13:12.793 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:13:12.793 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-10T19:13:12.793 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:1 2023-12-10T19:13:12.793 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:12.793 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:13:12.793 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-10T19:13:12.793 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:13:12.793 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:13:12.793 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:13:12.794 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:13:12.794 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:13:12.794 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:13:12.794 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:13:12.794 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:13:12.794 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:13:12.794 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:13:12.794 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-10T19:13:12.816 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:13:12.816 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:13:12.816 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:13:12.816 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_6 2023-12-10T19:13:12.924 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2023-12-10T19:13:12.925 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:13:12.925 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:13:12.927 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_6 2023-12-10T19:13:12.927 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:2 2023-12-10T19:13:12.927 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:13:12.927 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:13:12.928 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-10T19:13:12.928 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:13:12.928 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:13:12.928 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:13:12.928 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:13:12.928 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:13:12.928 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:13:12.928 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:13:12.928 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:13:12.928 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:13:12.928 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:13:12.928 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-10T19:13:12.928 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:2 2023-12-10T19:13:12.928 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:12.929 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:13:12.929 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-10T19:13:12.929 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:13:12.929 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:13:12.929 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:13:12.929 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:13:12.929 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:13:12.929 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:13:12.929 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:13:12.929 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:13:12.929 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:13:12.929 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:13:12.929 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-10T19:13:12.952 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:13:12.952 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:13:12.953 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:13:12.953 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_6 2023-12-10T19:13:13.074 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:13:13.075 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ status_result= "state": "Replaying" 2023-12-10T19:13:13.075 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep -i Replaying 2023-12-10T19:13:13.075 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo "state": "Replaying" 2023-12-10T19:13:13.075 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "state": "Replaying" 2023-12-10T19:13:13.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=started 2023-12-10T19:13:13.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test started = started 2023-12-10T19:13:13.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:13:13.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_6 2023-12-10T19:13:13.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:13:13.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:13:13.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:13.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_6 2023-12-10T19:13:13.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = journal ] 2023-12-10T19:13:13.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = snapshot ] 2023-12-10T19:13:13.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_6 2023-12-10T19:13:13.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:13:13.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:13:13.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:13.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_6 2023-12-10T19:13:13.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster2-mirror-image_6.status 2023-12-10T19:13:13.078 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2-mirror-image_6.status 2023-12-10T19:13:13.078 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:13:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_6.status 2023-12-10T19:13:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-image_6.status 2023-12-10T19:13:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-image_6.status 2023-12-10T19:13:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:13:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_6.status 2023-12-10T19:13:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mirror_image_snapshot cluster2 mirror image_6 2023-12-10T19:13:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:13:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_6 2023-12-10T19:13:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_6 2023-12-10T19:13:13.688 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Snapshot ID: 129 2023-12-10T19:13:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_6 /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_6.status 2023-12-10T19:13:13.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:13:13.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:13.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_6 2023-12-10T19:13:13.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_6.status 2023-12-10T19:13:13.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_6 --format xml 2023-12-10T19:13:13.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:13:13.758 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-10T19:13:13.760 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snapshot_id=129 2023-12-10T19:13:13.760 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:13:13.760 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.2 2023-12-10T19:13:13.962 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_6.status 2023-12-10T19:13:13.962 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:13.962 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:13.962 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_6 2023-12-10T19:13:13.962 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_6.status 2023-12-10T19:13:13.962 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2023-12-10T19:13:13.962 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:13:14.031 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:13:14.034 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=73 2023-12-10T19:13:14.034 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 129 = 73 2023-12-10T19:13:14.034 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.4 2023-12-10T19:13:14.435 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_6.status 2023-12-10T19:13:14.435 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:14.435 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:14.435 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_6 2023-12-10T19:13:14.436 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_6.status 2023-12-10T19:13:14.436 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2023-12-10T19:13:14.436 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:13:14.510 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:13:14.511 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:13:14.512 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:13:14.512 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:13:14.512 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:13:14.512 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:13:14.512 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 129 = 2023-12-10T19:13:14.512 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.8 2023-12-10T19:13:15.314 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_6.status 2023-12-10T19:13:15.314 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:15.314 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:15.314 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_6 2023-12-10T19:13:15.314 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_6.status 2023-12-10T19:13:15.314 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2023-12-10T19:13:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:13:15.386 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:13:15.386 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:13:15.386 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:13:15.386 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:13:15.387 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:13:15.387 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:13:15.387 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 129 = 2023-12-10T19:13:15.387 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1.6 2023-12-10T19:13:16.986 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_6.status 2023-12-10T19:13:16.986 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:16.987 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:16.987 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_6 2023-12-10T19:13:16.987 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_6.status 2023-12-10T19:13:16.987 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2023-12-10T19:13:16.988 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:13:17.049 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:13:17.051 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=129 2023-12-10T19:13:17.051 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 129 = 129 2023-12-10T19:13:17.051 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:13:17.051 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snap_present cluster1 mirror image_6 snap 2023-12-10T19:13:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_6 2023-12-10T19:13:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:13:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:13:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:13:18.052 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 info mirror/image_6@snap 2023-12-10T19:13:18.116 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_6': 2023-12-10T19:13:18.116 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:13:18.116 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:13:18.116 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 2 2023-12-10T19:13:18.116 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 10e825fc8e7a 2023-12-10T19:13:18.117 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.10e825fc8e7a 2023-12-10T19:13:18.117 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:13:18.117 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering, non-primary 2023-12-10T19:13:18.117 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:13:18.117 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:13:18.117 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:03:33 2023 2023-12-10T19:13:18.117 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:03:33 2023 2023-12-10T19:13:18.117 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:03:33 2023 2023-12-10T19:13:18.117 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: protected: False 2023-12-10T19:13:18.117 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:13:18.117 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:13:18.117 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: c1f7c734-b9ec-4996-95de-10380a5d720f 2023-12-10T19:13:18.117 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: false 2023-12-10T19:13:18.120 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:13:18.121 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror image_6 snap 2023-12-10T19:13:18.121 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:18.121 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_6 2023-12-10T19:13:18.121 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:13:18.121 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:13:18.121 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_6.export 2023-12-10T19:13:18.121 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_6.export 2023-12-10T19:13:18.121 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_6.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_6.export 2023-12-10T19:13:18.121 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export image_6@snap /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_6.export 2023-12-10T19:13:18.330 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:13:18.340 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export image_6@snap /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_6.export 2023-12-10T19:13:18.651 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:13:18.665 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_6.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_6.export 2023-12-10T19:13:18.740 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_6.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_6.export 2023-12-10T19:13:18.780 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:13:18.780 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_7 2023-12-10T19:13:18.780 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror image_7 snap 2023-12-10T19:13:18.780 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:13:18.780 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:18.780 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_7 2023-12-10T19:13:18.780 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:13:18.780 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create image_7@snap --debug-rbd=20 --debug-journaler=20 2023-12-10T19:13:19.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_started cluster1 mirror image_7 2023-12-10T19:13:19.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:19.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:19.699 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_7 2023-12-10T19:13:19.699 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_state cluster1 mirror image_7 started 2023-12-10T19:13:19.699 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:19.699 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:19.699 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_7 2023-12-10T19:13:19.699 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=started 2023-12-10T19:13:19.700 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:13:19.700 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:13:20.699 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_replay_state cluster1 mirror image_7 started 2023-12-10T19:13:20.699 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:20.700 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:20.700 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_7 2023-12-10T19:13:20.700 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=started 2023-12-10T19:13:20.700 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_result 2023-12-10T19:13:20.700 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=stopped 2023-12-10T19:13:20.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_7 2023-12-10T19:13:20.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_instance=cluster1 2023-12-10T19:13:20.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:13:20.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:20.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance=cluster1 2023-12-10T19:13:20.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loop_instance 2023-12-10T19:13:20.704 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:13:20.704 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep -i state 2023-12-10T19:13:20.704 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T19:13:20.704 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T19:13:20.704 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_7 2023-12-10T19:13:20.704 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T19:13:20.704 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:13:20.704 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:13:20.704 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T19:13:20.704 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:13:20.704 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:13:20.704 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:13:20.704 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:13:20.704 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:13:20.704 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:13:20.704 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:13:20.705 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:13:20.705 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:13:20.705 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:13:20.705 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T19:13:20.705 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T19:13:20.705 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:20.705 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:13:20.705 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T19:13:20.705 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:13:20.705 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:13:20.705 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:13:20.705 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:13:20.705 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:13:20.705 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:13:20.705 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:13:20.705 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:13:20.706 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:13:20.706 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:13:20.706 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T19:13:20.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:13:20.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:13:20.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:13:20.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_7 2023-12-10T19:13:20.838 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2023-12-10T19:13:20.838 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:13:20.838 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:13:20.840 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_7 2023-12-10T19:13:20.840 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:1 2023-12-10T19:13:20.840 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:13:20.840 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:13:20.840 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-10T19:13:20.840 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:13:20.841 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:13:20.841 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:13:20.841 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:13:20.841 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:13:20.841 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:13:20.841 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:13:20.841 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:13:20.841 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:13:20.841 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:13:20.841 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-10T19:13:20.841 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:1 2023-12-10T19:13:20.841 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:20.841 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:13:20.841 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-10T19:13:20.841 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:13:20.842 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:13:20.842 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:13:20.842 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:13:20.842 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:13:20.842 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:13:20.842 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:13:20.842 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:13:20.842 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:13:20.842 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:13:20.842 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-10T19:13:20.865 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:13:20.865 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:13:20.865 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:13:20.865 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_7 2023-12-10T19:13:20.983 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2023-12-10T19:13:20.984 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:13:20.984 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:13:20.986 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_7 2023-12-10T19:13:20.986 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:2 2023-12-10T19:13:20.986 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:13:20.986 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:13:20.986 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-10T19:13:20.986 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:13:20.986 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:13:20.987 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:13:20.987 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:13:20.987 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:13:20.987 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:13:20.987 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:13:20.987 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:13:20.987 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:13:20.987 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:13:20.987 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-10T19:13:20.987 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:2 2023-12-10T19:13:20.987 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:20.987 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:13:20.987 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-10T19:13:20.987 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:13:20.987 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:13:20.988 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:13:20.988 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:13:20.988 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:13:20.988 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:13:20.988 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:13:20.988 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:13:20.988 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:13:20.988 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:13:20.988 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-10T19:13:21.010 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:13:21.011 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:13:21.011 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:13:21.011 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_7 2023-12-10T19:13:21.119 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2023-12-10T19:13:21.120 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:13:21.120 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:13:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_7 2023-12-10T19:13:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:3 2023-12-10T19:13:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:13:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:13:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-10T19:13:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:3 2023-12-10T19:13:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:13:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:13:21.123 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:13:21.123 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:13:21.123 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 3 = cluster1:3 ] 2023-12-10T19:13:21.123 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:13:21.123 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:13:21.123 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=3 2023-12-10T19:13:21.123 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:13:21.123 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-10T19:13:21.123 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:3 2023-12-10T19:13:21.123 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:21.123 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:13:21.123 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-10T19:13:21.123 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:3 2023-12-10T19:13:21.123 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:13:21.123 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:13:21.123 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:13:21.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:13:21.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 3 = cluster1:3 ] 2023-12-10T19:13:21.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:13:21.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:13:21.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=3 2023-12-10T19:13:21.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:13:21.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-10T19:13:21.147 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T19:13:21.147 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T19:13:21.147 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T19:13:21.147 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_7 2023-12-10T19:13:21.271 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:13:21.271 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ status_result= "state": "Replaying" 2023-12-10T19:13:21.271 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo "state": "Replaying" 2023-12-10T19:13:21.271 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep -i Replaying 2023-12-10T19:13:21.272 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "state": "Replaying" 2023-12-10T19:13:21.273 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=started 2023-12-10T19:13:21.273 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test started = started 2023-12-10T19:13:21.273 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:13:21.273 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_7 2023-12-10T19:13:21.273 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:13:21.273 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:13:21.273 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:21.273 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_7 2023-12-10T19:13:21.273 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = journal ] 2023-12-10T19:13:21.273 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = snapshot ] 2023-12-10T19:13:21.273 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_7 2023-12-10T19:13:21.273 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:13:21.273 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:13:21.273 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:21.274 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_7 2023-12-10T19:13:21.274 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster2-mirror-image_7.status 2023-12-10T19:13:21.274 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2-mirror-image_7.status 2023-12-10T19:13:21.274 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:13:21.275 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_7.status 2023-12-10T19:13:21.275 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-image_7.status 2023-12-10T19:13:21.275 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-image_7.status 2023-12-10T19:13:21.275 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:13:21.275 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_7.status 2023-12-10T19:13:21.275 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mirror_image_snapshot cluster2 mirror image_7 2023-12-10T19:13:21.276 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:13:21.276 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:21.276 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_7 2023-12-10T19:13:21.276 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_7 2023-12-10T19:13:21.729 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Snapshot ID: 131 2023-12-10T19:13:21.742 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_7 /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_7.status 2023-12-10T19:13:21.743 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:13:21.743 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:21.743 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_7 2023-12-10T19:13:21.743 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_7.status 2023-12-10T19:13:21.743 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_7 --format xml 2023-12-10T19:13:21.743 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:13:21.797 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-10T19:13:21.800 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snapshot_id=131 2023-12-10T19:13:21.800 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:13:21.800 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.2 2023-12-10T19:13:22.001 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_7.status 2023-12-10T19:13:22.001 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:22.001 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:22.001 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_7 2023-12-10T19:13:22.001 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_7.status 2023-12-10T19:13:22.001 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2023-12-10T19:13:22.002 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:13:22.074 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:13:22.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=74 2023-12-10T19:13:22.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 131 = 74 2023-12-10T19:13:22.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.4 2023-12-10T19:13:22.478 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_7.status 2023-12-10T19:13:22.478 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:22.478 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:22.478 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_7 2023-12-10T19:13:22.478 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_7.status 2023-12-10T19:13:22.478 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2023-12-10T19:13:22.479 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:13:22.546 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:13:22.546 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:13:22.548 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:13:22.548 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:13:22.548 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:13:22.548 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:13:22.548 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 131 = 2023-12-10T19:13:22.548 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.8 2023-12-10T19:13:23.349 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_7.status 2023-12-10T19:13:23.350 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:23.350 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:23.350 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_7 2023-12-10T19:13:23.350 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_7.status 2023-12-10T19:13:23.350 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2023-12-10T19:13:23.350 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:13:23.418 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:13:23.418 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:13:23.420 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:13:23.420 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:13:23.420 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:13:23.420 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:13:23.420 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 131 = 2023-12-10T19:13:23.420 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1.6 2023-12-10T19:13:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_7.status 2023-12-10T19:13:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_7 2023-12-10T19:13:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_7.status 2023-12-10T19:13:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2023-12-10T19:13:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:13:25.094 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:13:25.098 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=131 2023-12-10T19:13:25.098 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 131 = 131 2023-12-10T19:13:25.098 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:13:25.098 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snap_present cluster1 mirror image_7 snap 2023-12-10T19:13:25.098 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:25.098 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:25.098 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_7 2023-12-10T19:13:25.098 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:13:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:13:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:13:26.098 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 info mirror/image_7@snap 2023-12-10T19:13:26.166 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_7': 2023-12-10T19:13:26.166 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:13:26.166 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:13:26.166 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 2 2023-12-10T19:13:26.167 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 117eb8917850 2023-12-10T19:13:26.167 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.117eb8917850 2023-12-10T19:13:26.167 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:13:26.167 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering, non-primary 2023-12-10T19:13:26.167 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:13:26.167 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:13:26.167 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:03:47 2023 2023-12-10T19:13:26.167 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:03:47 2023 2023-12-10T19:13:26.167 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:03:47 2023 2023-12-10T19:13:26.167 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: protected: False 2023-12-10T19:13:26.167 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:13:26.167 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:13:26.167 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: f8539d7e-11cd-4f60-9d51-e1da51589906 2023-12-10T19:13:26.167 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: false 2023-12-10T19:13:26.171 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:13:26.171 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror image_7 snap 2023-12-10T19:13:26.171 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:26.171 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_7 2023-12-10T19:13:26.171 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:13:26.171 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:13:26.171 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_7.export 2023-12-10T19:13:26.171 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_7.export 2023-12-10T19:13:26.171 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_7.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_7.export 2023-12-10T19:13:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export image_7@snap /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_7.export 2023-12-10T19:13:26.365 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:13:26.372 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export image_7@snap /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_7.export 2023-12-10T19:13:26.719 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:13:26.734 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_7.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_7.export 2023-12-10T19:13:26.809 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_7.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_7.export 2023-12-10T19:13:26.849 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:13:26.849 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_8 2023-12-10T19:13:26.849 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror image_8 snap 2023-12-10T19:13:26.849 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:13:26.849 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:26.849 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_8 2023-12-10T19:13:26.849 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:13:26.849 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create image_8@snap --debug-rbd=20 --debug-journaler=20 2023-12-10T19:13:27.336 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_started cluster1 mirror image_8 2023-12-10T19:13:27.337 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:27.337 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:27.337 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_8 2023-12-10T19:13:27.337 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_state cluster1 mirror image_8 started 2023-12-10T19:13:27.337 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:27.337 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:27.337 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_8 2023-12-10T19:13:27.337 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=started 2023-12-10T19:13:27.337 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:13:27.337 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:13:28.337 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_replay_state cluster1 mirror image_8 started 2023-12-10T19:13:28.338 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:28.338 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:28.338 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_8 2023-12-10T19:13:28.338 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=started 2023-12-10T19:13:28.338 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_result 2023-12-10T19:13:28.338 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=stopped 2023-12-10T19:13:28.339 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_8 2023-12-10T19:13:28.339 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ + local cluster_instance=cluster1 2023-12-10T19:13:28.339 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:13:28.339 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:grep -i state+ 2023-12-10T19:13:28.339 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:local cluster=cluster1 2023-12-10T19:13:28.339 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance=cluster1 2023-12-10T19:13:28.339 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loop_instance 2023-12-10T19:13:28.339 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:13:28.339 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T19:13:28.340 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T19:13:28.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_8 2023-12-10T19:13:28.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T19:13:28.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:13:28.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:13:28.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T19:13:28.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:13:28.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:13:28.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:13:28.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:13:28.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:13:28.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:13:28.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:13:28.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:13:28.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:13:28.342 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:13:28.342 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T19:13:28.342 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T19:13:28.342 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:28.342 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:13:28.342 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T19:13:28.342 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:13:28.342 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:13:28.342 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:13:28.342 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:13:28.342 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:13:28.342 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:13:28.342 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:13:28.342 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:13:28.342 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:13:28.342 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:13:28.343 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T19:13:28.368 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:13:28.368 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:13:28.368 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:13:28.368 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_8 2023-12-10T19:13:28.483 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2023-12-10T19:13:28.483 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:13:28.483 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:13:28.486 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_8 2023-12-10T19:13:28.486 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:1 2023-12-10T19:13:28.486 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:13:28.486 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:13:28.486 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-10T19:13:28.486 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:13:28.486 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:13:28.486 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:13:28.486 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:13:28.486 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:13:28.486 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:13:28.487 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:13:28.487 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:13:28.487 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:13:28.487 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:13:28.487 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-10T19:13:28.487 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:1 2023-12-10T19:13:28.487 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:28.487 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:13:28.487 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-10T19:13:28.487 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:13:28.487 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:13:28.487 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:13:28.488 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:13:28.488 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:13:28.488 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:13:28.488 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:13:28.488 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:13:28.488 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:13:28.488 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:13:28.489 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-10T19:13:28.509 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:13:28.510 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:13:28.510 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:13:28.510 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_8 2023-12-10T19:13:28.632 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:13:28.632 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ status_result= "state": "Replaying" 2023-12-10T19:13:28.632 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo "state": "Replaying" 2023-12-10T19:13:28.632 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep -i Replaying 2023-12-10T19:13:28.633 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "state": "Replaying" 2023-12-10T19:13:28.634 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=started 2023-12-10T19:13:28.634 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test started = started 2023-12-10T19:13:28.634 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:13:28.634 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_8 2023-12-10T19:13:28.634 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:13:28.634 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:13:28.634 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:28.634 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_8 2023-12-10T19:13:28.634 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = journal ] 2023-12-10T19:13:28.635 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = snapshot ] 2023-12-10T19:13:28.635 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_8 2023-12-10T19:13:28.635 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:13:28.635 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:13:28.635 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:28.635 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_8 2023-12-10T19:13:28.635 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster2-mirror-image_8.status 2023-12-10T19:13:28.635 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2-mirror-image_8.status 2023-12-10T19:13:28.635 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:13:28.636 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_8.status 2023-12-10T19:13:28.636 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-image_8.status 2023-12-10T19:13:28.636 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-image_8.status 2023-12-10T19:13:28.636 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:13:28.636 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_8.status 2023-12-10T19:13:28.637 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mirror_image_snapshot cluster2 mirror image_8 2023-12-10T19:13:28.637 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:13:28.637 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:28.637 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_8 2023-12-10T19:13:28.637 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_8 2023-12-10T19:13:29.364 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Snapshot ID: 133 2023-12-10T19:13:29.378 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_8 /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_8.status 2023-12-10T19:13:29.378 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:13:29.378 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:29.378 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_8 2023-12-10T19:13:29.378 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_8.status 2023-12-10T19:13:29.378 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_8 --format xml 2023-12-10T19:13:29.378 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:13:29.433 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-10T19:13:29.435 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snapshot_id=133 2023-12-10T19:13:29.435 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:13:29.435 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.2 2023-12-10T19:13:29.636 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_8.status 2023-12-10T19:13:29.637 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:29.637 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:29.637 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_8 2023-12-10T19:13:29.637 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_8.status 2023-12-10T19:13:29.637 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2023-12-10T19:13:29.637 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:13:29.720 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:13:29.720 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:13:29.722 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:13:29.722 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:13:29.722 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:13:29.723 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:13:29.723 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 133 = 2023-12-10T19:13:29.723 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.4 2023-12-10T19:13:30.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_8.status 2023-12-10T19:13:30.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:30.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:30.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_8 2023-12-10T19:13:30.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_8.status 2023-12-10T19:13:30.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2023-12-10T19:13:30.125 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:13:30.195 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:13:30.195 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:13:30.198 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:13:30.198 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:13:30.198 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:13:30.198 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:13:30.199 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 133 = 2023-12-10T19:13:30.199 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.8 2023-12-10T19:13:30.999 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_8.status 2023-12-10T19:13:30.999 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:31.000 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:31.000 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_8 2023-12-10T19:13:31.000 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_8.status 2023-12-10T19:13:31.000 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2023-12-10T19:13:31.000 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:13:31.073 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:13:31.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=133 2023-12-10T19:13:31.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 133 = 133 2023-12-10T19:13:31.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:13:31.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snap_present cluster1 mirror image_8 snap 2023-12-10T19:13:31.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:31.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:31.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_8 2023-12-10T19:13:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:13:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:13:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:13:32.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 info mirror/image_8@snap 2023-12-10T19:13:32.147 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_8': 2023-12-10T19:13:32.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:13:32.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:13:32.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 2 2023-12-10T19:13:32.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 12009f749fd5 2023-12-10T19:13:32.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.12009f749fd5 2023-12-10T19:13:32.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:13:32.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering, non-primary 2023-12-10T19:13:32.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:13:32.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:13:32.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:04:00 2023 2023-12-10T19:13:32.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:04:00 2023 2023-12-10T19:13:32.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:04:00 2023 2023-12-10T19:13:32.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: protected: False 2023-12-10T19:13:32.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:13:32.149 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:13:32.149 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: f24c77de-98cd-45a4-9355-6e2e919e8517 2023-12-10T19:13:32.149 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: false 2023-12-10T19:13:32.152 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:13:32.152 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror image_8 snap 2023-12-10T19:13:32.152 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:32.152 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_8 2023-12-10T19:13:32.152 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:13:32.152 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:13:32.152 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_8.export 2023-12-10T19:13:32.152 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_8.export 2023-12-10T19:13:32.152 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_8.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_8.export 2023-12-10T19:13:32.153 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export image_8@snap /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_8.export 2023-12-10T19:13:32.356 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:13:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export image_8@snap /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_8.export 2023-12-10T19:13:32.726 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:13:32.741 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_8.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_8.export 2023-12-10T19:13:32.820 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_8.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_8.export 2023-12-10T19:13:32.859 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:13:32.859 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_9 2023-12-10T19:13:32.859 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror image_9 snap 2023-12-10T19:13:32.859 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:13:32.860 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:32.860 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_9 2023-12-10T19:13:32.860 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:13:32.860 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create image_9@snap --debug-rbd=20 --debug-journaler=20 2023-12-10T19:13:33.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_started cluster1 mirror image_9 2023-12-10T19:13:33.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:33.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:33.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_9 2023-12-10T19:13:33.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_state cluster1 mirror image_9 started 2023-12-10T19:13:33.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:33.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:33.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_9 2023-12-10T19:13:33.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=started 2023-12-10T19:13:33.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:13:33.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:13:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_replay_state cluster1 mirror image_9 started 2023-12-10T19:13:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_9 2023-12-10T19:13:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=started 2023-12-10T19:13:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_result 2023-12-10T19:13:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=stopped 2023-12-10T19:13:34.662 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_9 2023-12-10T19:13:34.662 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_instance=cluster1 2023-12-10T19:13:34.662 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:13:34.662 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:34.662 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance=cluster1 2023-12-10T19:13:34.662 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loop_instance 2023-12-10T19:13:34.662 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:13:34.662 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep -i state 2023-12-10T19:13:34.664 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T19:13:34.664 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T19:13:34.664 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_9 2023-12-10T19:13:34.664 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T19:13:34.664 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:13:34.664 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:13:34.664 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T19:13:34.664 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:13:34.664 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:13:34.665 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:13:34.665 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:13:34.665 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:13:34.665 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:13:34.665 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:13:34.665 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:13:34.665 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:13:34.665 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:13:34.665 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T19:13:34.665 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T19:13:34.665 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:34.665 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:13:34.665 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T19:13:34.665 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:13:34.665 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:13:34.666 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:13:34.666 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:13:34.666 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:13:34.666 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:13:34.666 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:13:34.666 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:13:34.666 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:13:34.666 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:13:34.666 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T19:13:34.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:13:34.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:13:34.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:13:34.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_9 2023-12-10T19:13:34.799 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_9'] 2023-12-10T19:13:34.799 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:13:34.799 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:13:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_9 2023-12-10T19:13:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:1 2023-12-10T19:13:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:13:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:13:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-10T19:13:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:13:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:13:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:13:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:13:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:13:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:13:34.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:13:34.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:13:34.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:13:34.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:13:34.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-10T19:13:34.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:1 2023-12-10T19:13:34.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:34.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:13:34.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-10T19:13:34.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:13:34.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:13:34.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:13:34.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:13:34.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:13:34.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:13:34.803 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:13:34.803 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:13:34.803 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:13:34.803 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:13:34.803 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-10T19:13:34.825 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:13:34.826 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:13:34.826 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:13:34.826 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_9 2023-12-10T19:13:34.946 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_9'] 2023-12-10T19:13:34.946 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:13:34.946 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:13:34.949 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_9 2023-12-10T19:13:34.949 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:2 2023-12-10T19:13:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:13:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:13:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-10T19:13:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:13:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:13:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:13:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:13:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:13:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:13:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:13:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:13:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:13:34.952 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:13:34.952 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-10T19:13:34.952 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:2 2023-12-10T19:13:34.952 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:34.952 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:13:34.952 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-10T19:13:34.952 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:13:34.952 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:13:34.953 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:13:34.953 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:13:34.953 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:13:34.953 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:13:34.953 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:13:34.954 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:13:34.954 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:13:34.954 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:13:34.954 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-10T19:13:34.973 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:13:34.973 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:13:34.973 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:13:34.973 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_9 2023-12-10T19:13:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_9'] 2023-12-10T19:13:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:13:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:13:35.095 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_9 2023-12-10T19:13:35.095 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:3 2023-12-10T19:13:35.095 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:13:35.095 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:13:35.095 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-10T19:13:35.095 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:3 2023-12-10T19:13:35.095 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:13:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:13:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:13:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:13:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 3 = cluster1:3 ] 2023-12-10T19:13:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:13:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:13:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=3 2023-12-10T19:13:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:13:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-10T19:13:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:3 2023-12-10T19:13:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:13:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-10T19:13:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:3 2023-12-10T19:13:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:13:35.097 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:13:35.097 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:13:35.097 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:13:35.097 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 3 = cluster1:3 ] 2023-12-10T19:13:35.097 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:13:35.097 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:13:35.097 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=3 2023-12-10T19:13:35.097 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:13:35.097 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-10T19:13:35.115 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T19:13:35.115 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T19:13:35.115 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T19:13:35.115 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_9 2023-12-10T19:13:35.236 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:13:35.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ status_result= "state": "Replaying" 2023-12-10T19:13:35.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo "state": "Replaying" 2023-12-10T19:13:35.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep -i Replaying 2023-12-10T19:13:35.238 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "state": "Replaying" 2023-12-10T19:13:35.239 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=started 2023-12-10T19:13:35.239 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test started = started 2023-12-10T19:13:35.239 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:13:35.239 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_9 2023-12-10T19:13:35.239 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:13:35.239 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:13:35.239 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:35.240 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_9 2023-12-10T19:13:35.240 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = journal ] 2023-12-10T19:13:35.240 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = snapshot ] 2023-12-10T19:13:35.240 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_9 2023-12-10T19:13:35.240 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:13:35.240 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:13:35.240 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:35.240 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_9 2023-12-10T19:13:35.240 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster2-mirror-image_9.status 2023-12-10T19:13:35.240 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2-mirror-image_9.status 2023-12-10T19:13:35.240 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:13:35.241 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_9.status 2023-12-10T19:13:35.241 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-image_9.status 2023-12-10T19:13:35.241 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-image_9.status 2023-12-10T19:13:35.241 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:13:35.242 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_9.status 2023-12-10T19:13:35.242 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mirror_image_snapshot cluster2 mirror image_9 2023-12-10T19:13:35.242 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:13:35.242 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:35.242 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_9 2023-12-10T19:13:35.242 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_9 2023-12-10T19:13:36.338 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Snapshot ID: 135 2023-12-10T19:13:36.353 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_9 /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_9.status 2023-12-10T19:13:36.353 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:13:36.353 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:36.353 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_9 2023-12-10T19:13:36.353 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_9.status 2023-12-10T19:13:36.353 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_9 --format xml 2023-12-10T19:13:36.354 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:13:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-10T19:13:36.411 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snapshot_id=135 2023-12-10T19:13:36.411 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:13:36.411 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.2 2023-12-10T19:13:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_9.status 2023-12-10T19:13:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_9 2023-12-10T19:13:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_9.status 2023-12-10T19:13:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2023-12-10T19:13:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:13:36.685 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:13:36.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:13:36.688 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:13:36.688 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:13:36.688 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:13:36.688 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:13:36.688 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 135 = 2023-12-10T19:13:36.688 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.4 2023-12-10T19:13:37.089 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_9.status 2023-12-10T19:13:37.089 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:37.089 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:37.089 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_9 2023-12-10T19:13:37.089 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_9.status 2023-12-10T19:13:37.090 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2023-12-10T19:13:37.090 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:13:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:13:37.160 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:13:37.162 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:13:37.162 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:13:37.162 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:13:37.162 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:13:37.162 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 135 = 2023-12-10T19:13:37.162 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.8 2023-12-10T19:13:37.963 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_9.status 2023-12-10T19:13:37.963 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:37.963 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:37.964 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_9 2023-12-10T19:13:37.964 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_9.status 2023-12-10T19:13:37.964 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2023-12-10T19:13:37.964 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:13:38.034 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:13:38.036 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=135 2023-12-10T19:13:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 135 = 135 2023-12-10T19:13:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:13:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snap_present cluster1 mirror image_9 snap 2023-12-10T19:13:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_9 2023-12-10T19:13:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:13:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:13:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:13:39.038 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 info mirror/image_9@snap 2023-12-10T19:13:39.098 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_9': 2023-12-10T19:13:39.098 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:13:39.098 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:13:39.098 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 2 2023-12-10T19:13:39.098 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 10c059acc2c 2023-12-10T19:13:39.098 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.10c059acc2c 2023-12-10T19:13:39.098 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:13:39.098 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering, non-primary 2023-12-10T19:13:39.098 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:13:39.099 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:13:39.099 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:04:12 2023 2023-12-10T19:13:39.099 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:04:12 2023 2023-12-10T19:13:39.099 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:04:12 2023 2023-12-10T19:13:39.099 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: protected: False 2023-12-10T19:13:39.099 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:13:39.099 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:13:39.099 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: ab1975bd-7976-4b27-85c2-606054317fe6 2023-12-10T19:13:39.099 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: false 2023-12-10T19:13:39.102 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:13:39.102 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror image_9 snap 2023-12-10T19:13:39.102 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:39.102 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_9 2023-12-10T19:13:39.102 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:13:39.102 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:13:39.102 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_9.export 2023-12-10T19:13:39.102 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_9.export 2023-12-10T19:13:39.102 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_9.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_9.export 2023-12-10T19:13:39.103 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export image_9@snap /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_9.export 2023-12-10T19:13:39.317 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:13:39.327 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export image_9@snap /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_9.export 2023-12-10T19:13:39.641 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:13:39.656 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_9.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_9.export 2023-12-10T19:13:39.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_9.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_9.export 2023-12-10T19:13:39.772 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:13:39.772 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_10 2023-12-10T19:13:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror image_10 snap 2023-12-10T19:13:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:13:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_10 2023-12-10T19:13:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:13:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create image_10@snap --debug-rbd=20 --debug-journaler=20 2023-12-10T19:13:40.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_started cluster1 mirror image_10 2023-12-10T19:13:40.661 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:40.661 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:40.661 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_10 2023-12-10T19:13:40.661 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_state cluster1 mirror image_10 started 2023-12-10T19:13:40.661 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:40.661 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:40.661 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_10 2023-12-10T19:13:40.661 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=started 2023-12-10T19:13:40.661 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:13:40.661 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:13:41.662 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_replay_state cluster1 mirror image_10 started 2023-12-10T19:13:41.662 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:41.662 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:41.662 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_10 2023-12-10T19:13:41.662 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=started 2023-12-10T19:13:41.663 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_result 2023-12-10T19:13:41.663 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=stopped 2023-12-10T19:13:41.664 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_10 2023-12-10T19:13:41.664 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_instance=cluster1 2023-12-10T19:13:41.664 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:13:41.664 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:41.664 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ + local instance=cluster1 2023-12-10T19:13:41.664 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loop_instance 2023-12-10T19:13:41.664 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:13:41.664 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:grep -i state 2023-12-10T19:13:41.664 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T19:13:41.664 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T19:13:41.666 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_10 2023-12-10T19:13:41.666 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T19:13:41.666 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:13:41.666 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:13:41.666 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T19:13:41.666 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:13:41.667 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:13:41.667 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:13:41.667 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:13:41.667 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:13:41.667 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:13:41.667 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:13:41.667 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:13:41.667 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:13:41.667 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:13:41.667 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T19:13:41.667 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T19:13:41.667 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:41.667 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:13:41.668 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T19:13:41.668 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:13:41.668 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:13:41.668 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:13:41.668 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:13:41.668 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:13:41.668 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:13:41.668 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:13:41.668 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:13:41.668 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:13:41.668 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:13:41.668 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T19:13:41.689 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:13:41.689 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:13:41.689 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:13:41.689 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_10 2023-12-10T19:13:41.801 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2023-12-10T19:13:41.801 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:13:41.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:13:41.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_10 2023-12-10T19:13:41.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:1 2023-12-10T19:13:41.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:13:41.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:13:41.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-10T19:13:41.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:13:41.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:13:41.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:13:41.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:13:41.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:13:41.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:13:41.805 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:13:41.805 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:13:41.805 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:13:41.805 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:13:41.805 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-10T19:13:41.805 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:1 2023-12-10T19:13:41.805 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:41.805 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:13:41.805 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-10T19:13:41.805 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:13:41.805 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:13:41.805 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:13:41.805 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:13:41.805 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:13:41.805 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:13:41.806 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:13:41.806 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:13:41.806 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:13:41.806 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:13:41.806 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-10T19:13:41.824 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:13:41.824 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:13:41.824 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:13:41.824 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_10 2023-12-10T19:13:41.960 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:13:41.960 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ status_result= "state": "Replaying" 2023-12-10T19:13:41.960 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep -i Replaying 2023-12-10T19:13:41.960 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo "state": "Replaying" 2023-12-10T19:13:41.961 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "state": "Replaying" 2023-12-10T19:13:41.962 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=started 2023-12-10T19:13:41.962 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test started = started 2023-12-10T19:13:41.962 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:13:41.962 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_10 2023-12-10T19:13:41.963 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:13:41.963 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:13:41.963 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:41.963 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_10 2023-12-10T19:13:41.963 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = journal ] 2023-12-10T19:13:41.963 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = snapshot ] 2023-12-10T19:13:41.963 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_10 2023-12-10T19:13:41.963 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:13:41.963 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:13:41.963 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:41.963 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_10 2023-12-10T19:13:41.963 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster2-mirror-image_10.status 2023-12-10T19:13:41.963 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2-mirror-image_10.status 2023-12-10T19:13:41.963 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:13:41.964 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_10.status 2023-12-10T19:13:41.964 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-image_10.status 2023-12-10T19:13:41.964 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-image_10.status 2023-12-10T19:13:41.964 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:13:41.965 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_10.status 2023-12-10T19:13:41.965 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mirror_image_snapshot cluster2 mirror image_10 2023-12-10T19:13:41.965 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:13:41.965 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:41.965 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_10 2023-12-10T19:13:41.965 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_10 2023-12-10T19:13:42.676 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Snapshot ID: 137 2023-12-10T19:13:42.689 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_10 /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_10.status 2023-12-10T19:13:42.689 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:13:42.689 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:42.689 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_10 2023-12-10T19:13:42.689 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_10.status 2023-12-10T19:13:42.689 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_10 --format xml 2023-12-10T19:13:42.689 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:13:42.744 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-10T19:13:42.746 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snapshot_id=137 2023-12-10T19:13:42.746 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:13:42.747 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.2 2023-12-10T19:13:42.948 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_10.status 2023-12-10T19:13:42.948 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:42.948 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:42.948 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_10 2023-12-10T19:13:42.948 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_10.status 2023-12-10T19:13:42.948 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2023-12-10T19:13:42.948 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:13:43.018 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:13:43.021 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=77 2023-12-10T19:13:43.021 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 137 = 77 2023-12-10T19:13:43.021 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.4 2023-12-10T19:13:43.422 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_10.status 2023-12-10T19:13:43.422 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:43.422 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:43.422 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_10 2023-12-10T19:13:43.422 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_10.status 2023-12-10T19:13:43.422 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2023-12-10T19:13:43.423 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:13:43.491 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:13:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=77 2023-12-10T19:13:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 137 = 77 2023-12-10T19:13:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.8 2023-12-10T19:13:44.295 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_10.status 2023-12-10T19:13:44.295 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:44.295 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:44.295 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_10 2023-12-10T19:13:44.295 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_10.status 2023-12-10T19:13:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2023-12-10T19:13:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:13:44.367 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:13:44.367 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:13:44.369 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:13:44.369 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:13:44.369 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:13:44.369 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:13:44.370 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 137 = 2023-12-10T19:13:44.370 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1.6 2023-12-10T19:13:45.971 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_10.status 2023-12-10T19:13:45.971 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:45.971 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:45.971 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_10 2023-12-10T19:13:45.971 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_10.status 2023-12-10T19:13:45.971 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2023-12-10T19:13:45.971 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:13:46.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:13:46.040 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=137 2023-12-10T19:13:46.040 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 137 = 137 2023-12-10T19:13:46.040 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:13:46.040 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snap_present cluster1 mirror image_10 snap 2023-12-10T19:13:46.040 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:46.040 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:46.040 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_10 2023-12-10T19:13:46.040 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:13:46.040 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:13:46.041 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:13:47.041 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 info mirror/image_10@snap 2023-12-10T19:13:47.106 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_10': 2023-12-10T19:13:47.106 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:13:47.106 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:13:47.106 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 2 2023-12-10T19:13:47.106 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 10e88364f05c 2023-12-10T19:13:47.106 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.10e88364f05c 2023-12-10T19:13:47.107 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:13:47.107 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering, non-primary 2023-12-10T19:13:47.107 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:13:47.107 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:13:47.107 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:04:23 2023 2023-12-10T19:13:47.107 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:04:23 2023 2023-12-10T19:13:47.107 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:04:23 2023 2023-12-10T19:13:47.107 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: protected: False 2023-12-10T19:13:47.107 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:13:47.108 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:13:47.108 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: 5e79b28d-acd2-4fbc-9a84-6c7b4c7dc476 2023-12-10T19:13:47.108 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: false 2023-12-10T19:13:47.111 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:13:47.111 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror image_10 snap 2023-12-10T19:13:47.111 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:47.111 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_10 2023-12-10T19:13:47.112 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:13:47.112 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:13:47.112 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_10.export 2023-12-10T19:13:47.112 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_10.export 2023-12-10T19:13:47.112 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_10.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_10.export 2023-12-10T19:13:47.112 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export image_10@snap /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_10.export 2023-12-10T19:13:47.333 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:13:47.342 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export image_10@snap /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_10.export 2023-12-10T19:13:47.691 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:13:47.705 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_10.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_10.export 2023-12-10T19:13:47.788 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_10.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_10.export 2023-12-10T19:13:47.828 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:13:47.828 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_11 2023-12-10T19:13:47.828 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror image_11 snap 2023-12-10T19:13:47.828 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:13:47.828 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:47.828 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_11 2023-12-10T19:13:47.828 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:13:47.828 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create image_11@snap --debug-rbd=20 --debug-journaler=20 2023-12-10T19:13:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_started cluster1 mirror image_11 2023-12-10T19:13:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_11 2023-12-10T19:13:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_state cluster1 mirror image_11 started 2023-12-10T19:13:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_11 2023-12-10T19:13:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=started 2023-12-10T19:13:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:13:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:13:49.665 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_replay_state cluster1 mirror image_11 started 2023-12-10T19:13:49.665 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_11 2023-12-10T19:13:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=started 2023-12-10T19:13:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_result 2023-12-10T19:13:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=stopped 2023-12-10T19:13:49.667 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemons cluster1 rbd mirror+ status mirror/image_11 2023-12-10T19:13:49.667 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:grep+ -i statelocal 2023-12-10T19:13:49.667 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: cluster_instance=cluster1 2023-12-10T19:13:49.667 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:13:49.667 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:49.667 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance=cluster1 2023-12-10T19:13:49.667 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loop_instance 2023-12-10T19:13:49.667 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:13:49.668 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T19:13:49.668 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T19:13:49.669 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_11 2023-12-10T19:13:49.669 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T19:13:49.669 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:13:49.669 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:13:49.669 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T19:13:49.670 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:13:49.670 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:13:49.670 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:13:49.670 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:13:49.670 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:13:49.670 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:13:49.670 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:13:49.670 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:13:49.670 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:13:49.670 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:13:49.670 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T19:13:49.670 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T19:13:49.670 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:49.670 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:13:49.670 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T19:13:49.671 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:13:49.671 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:13:49.671 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:13:49.671 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:13:49.671 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:13:49.671 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:13:49.671 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:13:49.671 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:13:49.671 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:13:49.671 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:13:49.671 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T19:13:49.692 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:13:49.692 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:13:49.692 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:13:49.693 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_11 2023-12-10T19:13:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2023-12-10T19:13:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:13:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:13:49.811 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_11 2023-12-10T19:13:49.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:1 2023-12-10T19:13:49.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:13:49.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:13:49.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-10T19:13:49.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:13:49.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:13:49.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:13:49.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:13:49.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:13:49.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:13:49.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:13:49.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:13:49.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:13:49.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:13:49.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-10T19:13:49.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:1 2023-12-10T19:13:49.813 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:49.813 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:13:49.813 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-10T19:13:49.813 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:13:49.813 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:13:49.813 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:13:49.813 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:13:49.813 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:13:49.813 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:13:49.813 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:13:49.813 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:13:49.813 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:13:49.813 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:13:49.813 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-10T19:13:49.830 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:13:49.830 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:13:49.830 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:13:49.830 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_11 2023-12-10T19:13:49.950 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2023-12-10T19:13:49.951 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:13:49.951 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:13:49.953 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_11 2023-12-10T19:13:49.953 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:2 2023-12-10T19:13:49.953 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:13:49.953 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:13:49.953 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-10T19:13:49.953 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:13:49.953 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:13:49.953 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:13:49.953 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:13:49.954 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:13:49.954 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:13:49.954 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:13:49.954 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:13:49.954 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:13:49.954 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:13:49.954 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-10T19:13:49.954 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:2 2023-12-10T19:13:49.954 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:49.954 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:13:49.954 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-10T19:13:49.954 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:13:49.954 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:13:49.954 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:13:49.954 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:13:49.955 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:13:49.955 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:13:49.955 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:13:49.955 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:13:49.955 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:13:49.955 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:13:49.955 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-10T19:13:49.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:13:49.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:13:49.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:13:49.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_11 2023-12-10T19:13:50.110 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2023-12-10T19:13:50.110 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:13:50.110 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:13:50.113 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_11 2023-12-10T19:13:50.113 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:3 2023-12-10T19:13:50.113 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:13:50.113 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:13:50.113 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-10T19:13:50.113 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:3 2023-12-10T19:13:50.113 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:13:50.113 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:13:50.113 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:13:50.113 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:13:50.114 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 3 = cluster1:3 ] 2023-12-10T19:13:50.114 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:13:50.114 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:13:50.114 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=3 2023-12-10T19:13:50.114 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:13:50.114 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-10T19:13:50.114 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:3 2023-12-10T19:13:50.114 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:50.114 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:13:50.114 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-10T19:13:50.114 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:3 2023-12-10T19:13:50.114 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:13:50.114 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:13:50.114 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:13:50.115 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:13:50.115 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 3 = cluster1:3 ] 2023-12-10T19:13:50.115 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:13:50.115 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:13:50.115 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=3 2023-12-10T19:13:50.115 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:13:50.115 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-10T19:13:50.136 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T19:13:50.136 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T19:13:50.136 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T19:13:50.136 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_11 2023-12-10T19:13:50.256 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:13:50.256 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ status_result= "state": "Replaying" 2023-12-10T19:13:50.256 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo "state": "Replaying" 2023-12-10T19:13:50.256 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep -i Replaying 2023-12-10T19:13:50.257 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "state": "Replaying" 2023-12-10T19:13:50.259 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=started 2023-12-10T19:13:50.259 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test started = started 2023-12-10T19:13:50.259 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:13:50.259 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_11 2023-12-10T19:13:50.259 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:13:50.259 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:13:50.259 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:50.259 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_11 2023-12-10T19:13:50.259 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = journal ] 2023-12-10T19:13:50.260 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = snapshot ] 2023-12-10T19:13:50.260 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_11 2023-12-10T19:13:50.260 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:13:50.260 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:13:50.260 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:50.260 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_11 2023-12-10T19:13:50.260 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster2-mirror-image_11.status 2023-12-10T19:13:50.260 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2-mirror-image_11.status 2023-12-10T19:13:50.260 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:13:50.261 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_11.status 2023-12-10T19:13:50.261 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-image_11.status 2023-12-10T19:13:50.261 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-image_11.status 2023-12-10T19:13:50.261 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:13:50.262 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_11.status 2023-12-10T19:13:50.262 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mirror_image_snapshot cluster2 mirror image_11 2023-12-10T19:13:50.262 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:13:50.262 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:50.262 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_11 2023-12-10T19:13:50.262 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_11 2023-12-10T19:13:50.690 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Snapshot ID: 139 2023-12-10T19:13:50.705 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_11 /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_11.status 2023-12-10T19:13:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:13:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_11 2023-12-10T19:13:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_11.status 2023-12-10T19:13:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_11 --format xml 2023-12-10T19:13:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:13:50.759 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-10T19:13:50.761 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snapshot_id=139 2023-12-10T19:13:50.761 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:13:50.761 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.2 2023-12-10T19:13:50.963 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_11.status 2023-12-10T19:13:50.964 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:50.964 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:50.964 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_11 2023-12-10T19:13:50.964 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_11.status 2023-12-10T19:13:50.964 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2023-12-10T19:13:50.964 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:13:51.027 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:13:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=78 2023-12-10T19:13:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 139 = 78 2023-12-10T19:13:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.4 2023-12-10T19:13:51.430 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_11.status 2023-12-10T19:13:51.431 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:51.431 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:51.431 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_11 2023-12-10T19:13:51.431 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_11.status 2023-12-10T19:13:51.431 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2023-12-10T19:13:51.432 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:13:51.531 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:13:51.531 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:13:51.533 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:13:51.533 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:13:51.533 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:13:51.534 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:13:51.534 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 139 = 2023-12-10T19:13:51.534 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.8 2023-12-10T19:13:52.335 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_11.status 2023-12-10T19:13:52.335 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:52.335 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:52.335 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_11 2023-12-10T19:13:52.335 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_11.status 2023-12-10T19:13:52.335 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2023-12-10T19:13:52.335 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:13:52.403 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:13:52.403 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:13:52.405 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:13:52.405 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:13:52.405 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:13:52.406 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:13:52.406 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 139 = 2023-12-10T19:13:52.406 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1.6 2023-12-10T19:13:54.007 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_11.status 2023-12-10T19:13:54.007 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:54.007 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:54.007 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_11 2023-12-10T19:13:54.007 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_11.status 2023-12-10T19:13:54.007 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2023-12-10T19:13:54.008 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:13:54.075 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:13:54.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=139 2023-12-10T19:13:54.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 139 = 139 2023-12-10T19:13:54.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:13:54.078 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snap_present cluster1 mirror image_11 snap 2023-12-10T19:13:54.078 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:54.078 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:54.078 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_11 2023-12-10T19:13:54.078 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:13:54.078 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:13:54.078 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:13:54.867 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.8561953834319013) > weight (0.25) 2023-12-10T19:13:54.867 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.9417521679619024) > weight (0.25) 2023-12-10T19:13:54.867 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.9208558758003238) > weight (0.25) 2023-12-10T19:13:54.867 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.9290784920936489) > weight (0.25) 2023-12-10T19:13:54.867 INFO:tasks.rbd_mirror_thrash:waiting for 91 secs before thrashing 2023-12-10T19:13:55.078 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 info mirror/image_11@snap 2023-12-10T19:13:55.149 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_11': 2023-12-10T19:13:55.149 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:13:55.149 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:13:55.149 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 2 2023-12-10T19:13:55.149 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 117e1ceff87d 2023-12-10T19:13:55.149 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.117e1ceff87d 2023-12-10T19:13:55.149 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:13:55.149 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering, non-primary 2023-12-10T19:13:55.149 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:13:55.149 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:13:55.149 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:04:30 2023 2023-12-10T19:13:55.150 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:04:30 2023 2023-12-10T19:13:55.150 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:04:30 2023 2023-12-10T19:13:55.150 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: protected: False 2023-12-10T19:13:55.150 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:13:55.150 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:13:55.150 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: d372b711-4545-4edb-a533-b051b75e68e3 2023-12-10T19:13:55.150 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: false 2023-12-10T19:13:55.153 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:13:55.153 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror image_11 snap 2023-12-10T19:13:55.153 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:55.153 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_11 2023-12-10T19:13:55.153 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:13:55.154 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:13:55.154 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_11.export 2023-12-10T19:13:55.154 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_11.export 2023-12-10T19:13:55.154 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_11.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_11.export 2023-12-10T19:13:55.154 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export image_11@snap /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_11.export 2023-12-10T19:13:55.360 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:13:55.368 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export image_11@snap /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_11.export 2023-12-10T19:13:55.700 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:13:55.715 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_11.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_11.export 2023-12-10T19:13:55.792 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_11.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_11.export 2023-12-10T19:13:55.832 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:13:55.832 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_12 2023-12-10T19:13:55.832 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror image_12 snap 2023-12-10T19:13:55.832 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:13:55.833 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:55.833 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_12 2023-12-10T19:13:55.833 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:13:55.833 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create image_12@snap --debug-rbd=20 --debug-journaler=20 2023-12-10T19:13:56.342 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_started cluster1 mirror image_12 2023-12-10T19:13:56.342 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:56.342 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:56.342 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_12 2023-12-10T19:13:56.342 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_state cluster1 mirror image_12 started 2023-12-10T19:13:56.343 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:56.343 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:56.343 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_12 2023-12-10T19:13:56.343 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=started 2023-12-10T19:13:56.343 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:13:56.343 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:13:57.344 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_replay_state cluster1 mirror image_12 started 2023-12-10T19:13:57.344 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:57.344 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:57.344 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_12 2023-12-10T19:13:57.344 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=started 2023-12-10T19:13:57.344 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_result 2023-12-10T19:13:57.344 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=stopped 2023-12-10T19:13:57.345 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_12 2023-12-10T19:13:57.345 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ + local cluster_instance=cluster1 2023-12-10T19:13:57.345 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:13:57.345 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:grep -i state 2023-12-10T19:13:57.345 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:57.345 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance=cluster1 2023-12-10T19:13:57.345 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loop_instance 2023-12-10T19:13:57.345 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:13:57.346 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T19:13:57.346 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T19:13:57.347 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_12 2023-12-10T19:13:57.347 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T19:13:57.350 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:13:57.350 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:13:57.350 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T19:13:57.350 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:13:57.350 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:13:57.350 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:13:57.350 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:13:57.350 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:13:57.350 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:13:57.350 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:13:57.351 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:13:57.351 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:13:57.351 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:13:57.351 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T19:13:57.351 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T19:13:57.351 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:57.351 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:13:57.351 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T19:13:57.351 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:13:57.352 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:13:57.352 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:13:57.352 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:13:57.352 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:13:57.352 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:13:57.352 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:13:57.352 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:13:57.352 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:13:57.352 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:13:57.352 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T19:13:57.373 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:13:57.374 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:13:57.374 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:13:57.374 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_12 2023-12-10T19:13:57.488 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2023-12-10T19:13:57.488 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:13:57.488 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:13:57.490 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_12 2023-12-10T19:13:57.493 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:1 2023-12-10T19:13:57.493 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:13:57.493 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:13:57.493 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-10T19:13:57.493 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:13:57.493 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:13:57.494 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:13:57.494 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:13:57.494 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:13:57.494 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:13:57.494 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:13:57.494 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:13:57.494 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:13:57.494 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:13:57.494 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-10T19:13:57.494 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:1 2023-12-10T19:13:57.494 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:57.494 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:13:57.494 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-10T19:13:57.494 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:13:57.494 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:13:57.495 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:13:57.495 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:13:57.495 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:13:57.495 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:13:57.495 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:13:57.495 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:13:57.495 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:13:57.495 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:13:57.495 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-10T19:13:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:13:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:13:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:13:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_12 2023-12-10T19:13:57.647 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:13:57.648 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ status_result= "state": "Replaying" 2023-12-10T19:13:57.648 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo "state": "Replaying" 2023-12-10T19:13:57.648 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep -i Replaying 2023-12-10T19:13:57.649 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "state": "Replaying" 2023-12-10T19:13:57.650 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=started 2023-12-10T19:13:57.650 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test started = started 2023-12-10T19:13:57.650 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:13:57.650 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_12 2023-12-10T19:13:57.650 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:13:57.650 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:13:57.650 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:57.650 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_12 2023-12-10T19:13:57.650 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = journal ] 2023-12-10T19:13:57.650 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = snapshot ] 2023-12-10T19:13:57.650 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_12 2023-12-10T19:13:57.650 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:13:57.651 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:13:57.651 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:57.651 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_12 2023-12-10T19:13:57.651 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster2-mirror-image_12.status 2023-12-10T19:13:57.651 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2-mirror-image_12.status 2023-12-10T19:13:57.651 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:13:57.652 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_12.status 2023-12-10T19:13:57.652 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-image_12.status 2023-12-10T19:13:57.652 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-image_12.status 2023-12-10T19:13:57.652 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:13:57.653 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_12.status 2023-12-10T19:13:57.653 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mirror_image_snapshot cluster2 mirror image_12 2023-12-10T19:13:57.653 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:13:57.653 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:57.653 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_12 2023-12-10T19:13:57.653 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_12 2023-12-10T19:13:58.375 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Snapshot ID: 141 2023-12-10T19:13:58.391 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_12 /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_12.status 2023-12-10T19:13:58.391 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:13:58.391 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:58.391 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_12 2023-12-10T19:13:58.391 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_12.status 2023-12-10T19:13:58.391 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_12 --format xml 2023-12-10T19:13:58.391 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:13:58.446 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-10T19:13:58.448 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snapshot_id=141 2023-12-10T19:13:58.448 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:13:58.448 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.2 2023-12-10T19:13:58.649 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_12.status 2023-12-10T19:13:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_12 2023-12-10T19:13:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_12.status 2023-12-10T19:13:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2023-12-10T19:13:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:13:58.724 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:13:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:13:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:13:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:13:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:13:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:13:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 141 = 2023-12-10T19:13:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.4 2023-12-10T19:13:59.129 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_12.status 2023-12-10T19:13:59.129 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:13:59.129 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:13:59.130 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_12 2023-12-10T19:13:59.130 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_12.status 2023-12-10T19:13:59.130 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2023-12-10T19:13:59.130 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:13:59.199 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:13:59.200 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:13:59.202 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:13:59.202 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:13:59.202 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:13:59.202 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:13:59.202 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 141 = 2023-12-10T19:13:59.202 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.8 2023-12-10T19:14:00.004 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_12.status 2023-12-10T19:14:00.004 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:00.004 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:00.004 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_12 2023-12-10T19:14:00.004 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_12.status 2023-12-10T19:14:00.004 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2023-12-10T19:14:00.004 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:14:00.075 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:14:00.078 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=141 2023-12-10T19:14:00.078 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 141 = 141 2023-12-10T19:14:00.078 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:14:00.078 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snap_present cluster1 mirror image_12 snap 2023-12-10T19:14:00.078 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:00.078 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:00.078 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_12 2023-12-10T19:14:00.078 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:14:00.078 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:14:00.078 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:14:01.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 info mirror/image_12@snap 2023-12-10T19:14:01.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_12': 2023-12-10T19:14:01.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:14:01.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:14:01.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 2 2023-12-10T19:14:01.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 1200861b4139 2023-12-10T19:14:01.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.1200861b4139 2023-12-10T19:14:01.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:14:01.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering, non-primary 2023-12-10T19:14:01.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:14:01.141 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:14:01.141 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:04:41 2023 2023-12-10T19:14:01.141 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:04:41 2023 2023-12-10T19:14:01.141 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:04:41 2023 2023-12-10T19:14:01.141 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: protected: False 2023-12-10T19:14:01.141 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:14:01.141 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:14:01.141 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: f2b859e8-6dbd-4781-91cf-11df1eb8a7a0 2023-12-10T19:14:01.141 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: false 2023-12-10T19:14:01.145 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:14:01.145 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror image_12 snap 2023-12-10T19:14:01.145 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:01.145 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_12 2023-12-10T19:14:01.146 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:14:01.146 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:14:01.146 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_12.export 2023-12-10T19:14:01.146 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_12.export 2023-12-10T19:14:01.146 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_12.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_12.export 2023-12-10T19:14:01.146 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export image_12@snap /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_12.export 2023-12-10T19:14:01.344 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:14:01.351 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export image_12@snap /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_12.export 2023-12-10T19:14:01.683 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:14:01.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_12.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_12.export 2023-12-10T19:14:01.773 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_12.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_12.export 2023-12-10T19:14:01.813 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:14:01.814 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_13 2023-12-10T19:14:01.814 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror image_13 snap 2023-12-10T19:14:01.814 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:14:01.814 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:01.814 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_13 2023-12-10T19:14:01.814 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:14:01.814 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create image_13@snap --debug-rbd=20 --debug-journaler=20 2023-12-10T19:14:02.375 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_started cluster1 mirror image_13 2023-12-10T19:14:02.375 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:02.375 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:02.376 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_13 2023-12-10T19:14:02.376 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_state cluster1 mirror image_13 started 2023-12-10T19:14:02.376 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:02.376 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:02.376 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_13 2023-12-10T19:14:02.376 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=started 2023-12-10T19:14:02.376 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:14:02.376 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:14:03.376 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_replay_state cluster1 mirror image_13 started 2023-12-10T19:14:03.377 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:03.377 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:03.377 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_13 2023-12-10T19:14:03.377 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=started 2023-12-10T19:14:03.377 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_result 2023-12-10T19:14:03.377 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=stopped 2023-12-10T19:14:03.378 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_13 2023-12-10T19:14:03.378 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_instance=cluster1 2023-12-10T19:14:03.378 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:14:03.378 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:03.378 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance=cluster1 2023-12-10T19:14:03.378 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ + local loop_instance 2023-12-10T19:14:03.378 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:14:03.378 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:grep -i state 2023-12-10T19:14:03.378 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T19:14:03.379 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T19:14:03.380 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_13 2023-12-10T19:14:03.380 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T19:14:03.380 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:14:03.380 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:14:03.380 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T19:14:03.380 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:14:03.380 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:14:03.380 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:14:03.380 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:14:03.380 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:14:03.381 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:14:03.381 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:14:03.381 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:14:03.381 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:14:03.381 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:14:03.381 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T19:14:03.381 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T19:14:03.381 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:03.381 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:14:03.381 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T19:14:03.381 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:14:03.381 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:14:03.381 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:14:03.381 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:14:03.381 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:14:03.381 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:14:03.382 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:14:03.382 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:14:03.382 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:14:03.382 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:14:03.382 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T19:14:03.405 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:14:03.405 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:14:03.405 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:14:03.405 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_13 2023-12-10T19:14:03.517 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2023-12-10T19:14:03.517 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:14:03.517 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:14:03.520 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_13 2023-12-10T19:14:03.520 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:1 2023-12-10T19:14:03.520 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:14:03.520 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:14:03.520 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-10T19:14:03.520 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:14:03.520 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:14:03.520 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:14:03.520 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:14:03.520 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:14:03.521 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:14:03.521 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:14:03.521 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:14:03.521 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:14:03.521 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:14:03.521 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-10T19:14:03.521 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:1 2023-12-10T19:14:03.521 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:03.521 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:14:03.521 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-10T19:14:03.521 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:14:03.521 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:14:03.521 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:14:03.521 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:14:03.521 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:14:03.521 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:14:03.522 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:14:03.522 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:14:03.522 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:14:03.522 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:14:03.522 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-10T19:14:03.539 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:14:03.540 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:14:03.540 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:14:03.540 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_13 2023-12-10T19:14:03.664 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2023-12-10T19:14:03.664 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:14:03.665 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:14:03.668 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_13 2023-12-10T19:14:03.668 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:2 2023-12-10T19:14:03.668 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:14:03.668 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:14:03.668 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-10T19:14:03.668 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:14:03.668 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:14:03.668 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:14:03.668 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:14:03.668 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:14:03.668 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:14:03.668 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:14:03.669 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:14:03.669 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:14:03.669 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:14:03.669 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-10T19:14:03.669 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:2 2023-12-10T19:14:03.669 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:03.669 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:14:03.669 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-10T19:14:03.669 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:14:03.669 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:14:03.669 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:14:03.669 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:14:03.669 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:14:03.669 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:14:03.669 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:14:03.669 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:14:03.670 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:14:03.670 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:14:03.670 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-10T19:14:03.690 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:14:03.691 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:14:03.691 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:14:03.691 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_13 2023-12-10T19:14:03.798 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2023-12-10T19:14:03.799 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:14:03.799 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:14:03.803 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_13 2023-12-10T19:14:03.803 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:3 2023-12-10T19:14:03.803 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:14:03.803 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:14:03.803 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-10T19:14:03.803 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:3 2023-12-10T19:14:03.803 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:14:03.803 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:14:03.803 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:14:03.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:14:03.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 3 = cluster1:3 ] 2023-12-10T19:14:03.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:14:03.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:14:03.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=3 2023-12-10T19:14:03.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:14:03.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-10T19:14:03.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:3 2023-12-10T19:14:03.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:03.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:14:03.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-10T19:14:03.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:3 2023-12-10T19:14:03.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:14:03.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:14:03.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:14:03.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:14:03.805 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 3 = cluster1:3 ] 2023-12-10T19:14:03.805 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:14:03.805 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:14:03.805 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=3 2023-12-10T19:14:03.805 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:14:03.805 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-10T19:14:03.827 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T19:14:03.827 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T19:14:03.827 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T19:14:03.828 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_13 2023-12-10T19:14:03.953 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:14:03.953 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ status_result= "state": "Replaying" 2023-12-10T19:14:03.954 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ + grep -i Replaying 2023-12-10T19:14:03.954 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:echo "state": "Replaying" 2023-12-10T19:14:03.954 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "state": "Replaying" 2023-12-10T19:14:03.955 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=started 2023-12-10T19:14:03.955 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test started = started 2023-12-10T19:14:03.955 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:14:03.955 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_13 2023-12-10T19:14:03.955 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:14:03.955 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:14:03.956 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:03.956 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_13 2023-12-10T19:14:03.956 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = journal ] 2023-12-10T19:14:03.956 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = snapshot ] 2023-12-10T19:14:03.956 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_13 2023-12-10T19:14:03.956 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:14:03.956 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:14:03.956 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:03.956 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_13 2023-12-10T19:14:03.956 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster2-mirror-image_13.status 2023-12-10T19:14:03.956 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2-mirror-image_13.status 2023-12-10T19:14:03.956 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:14:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_13.status 2023-12-10T19:14:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-image_13.status 2023-12-10T19:14:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-image_13.status 2023-12-10T19:14:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:14:03.958 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_13.status 2023-12-10T19:14:03.958 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mirror_image_snapshot cluster2 mirror image_13 2023-12-10T19:14:03.958 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:14:03.958 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:03.958 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_13 2023-12-10T19:14:03.958 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_13 2023-12-10T19:14:04.394 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Snapshot ID: 143 2023-12-10T19:14:04.408 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_13 /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_13.status 2023-12-10T19:14:04.408 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:14:04.408 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:04.408 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_13 2023-12-10T19:14:04.408 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_13.status 2023-12-10T19:14:04.408 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_13 --format xml 2023-12-10T19:14:04.408 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:14:04.463 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-10T19:14:04.466 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snapshot_id=143 2023-12-10T19:14:04.466 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:14:04.466 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.2 2023-12-10T19:14:04.667 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_13.status 2023-12-10T19:14:04.667 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:04.667 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:04.667 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_13 2023-12-10T19:14:04.667 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_13.status 2023-12-10T19:14:04.668 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2023-12-10T19:14:04.668 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:14:04.736 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:14:04.736 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:14:04.738 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:14:04.738 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:14:04.738 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:14:04.739 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:14:04.739 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 143 = 2023-12-10T19:14:04.739 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.4 2023-12-10T19:14:05.139 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_13.status 2023-12-10T19:14:05.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:05.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:05.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_13 2023-12-10T19:14:05.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_13.status 2023-12-10T19:14:05.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2023-12-10T19:14:05.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:14:05.207 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:14:05.207 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:14:05.209 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:14:05.209 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:14:05.209 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:14:05.209 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:14:05.209 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 143 = 2023-12-10T19:14:05.209 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.8 2023-12-10T19:14:06.011 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_13.status 2023-12-10T19:14:06.011 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:06.011 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:06.011 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_13 2023-12-10T19:14:06.011 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_13.status 2023-12-10T19:14:06.011 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2023-12-10T19:14:06.011 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:14:06.082 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:14:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=143 2023-12-10T19:14:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 143 = 143 2023-12-10T19:14:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:14:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snap_present cluster1 mirror image_13 snap 2023-12-10T19:14:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_13 2023-12-10T19:14:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:14:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:14:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:14:07.086 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 info mirror/image_13@snap 2023-12-10T19:14:07.153 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_13': 2023-12-10T19:14:07.153 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:14:07.153 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:14:07.153 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 2 2023-12-10T19:14:07.153 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 10c04303c784 2023-12-10T19:14:07.153 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.10c04303c784 2023-12-10T19:14:07.154 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:14:07.154 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering, non-primary 2023-12-10T19:14:07.154 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:14:07.154 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:14:07.154 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:04:53 2023 2023-12-10T19:14:07.154 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:04:53 2023 2023-12-10T19:14:07.154 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:04:53 2023 2023-12-10T19:14:07.154 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: protected: False 2023-12-10T19:14:07.154 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:14:07.154 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:14:07.154 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: 9cd9cf4d-71e8-4901-8340-6c85f5a313e8 2023-12-10T19:14:07.154 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: false 2023-12-10T19:14:07.158 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:14:07.158 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror image_13 snap 2023-12-10T19:14:07.158 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:07.158 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_13 2023-12-10T19:14:07.158 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:14:07.158 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:14:07.159 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_13.export 2023-12-10T19:14:07.159 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_13.export 2023-12-10T19:14:07.159 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_13.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_13.export 2023-12-10T19:14:07.159 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export image_13@snap /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_13.export 2023-12-10T19:14:07.363 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:14:07.372 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export image_13@snap /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_13.export 2023-12-10T19:14:07.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:14:07.699 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_13.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_13.export 2023-12-10T19:14:07.775 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_13.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_13.export 2023-12-10T19:14:07.814 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:14:07.815 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_14 2023-12-10T19:14:07.815 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror image_14 snap 2023-12-10T19:14:07.815 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:14:07.815 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:07.815 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_14 2023-12-10T19:14:07.815 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:14:07.815 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create image_14@snap --debug-rbd=20 --debug-journaler=20 2023-12-10T19:14:08.668 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_started cluster1 mirror image_14 2023-12-10T19:14:08.668 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:08.668 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:08.668 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_14 2023-12-10T19:14:08.668 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_state cluster1 mirror image_14 started 2023-12-10T19:14:08.668 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:08.669 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:08.669 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_14 2023-12-10T19:14:08.669 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=started 2023-12-10T19:14:08.669 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:14:08.669 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:14:09.669 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_replay_state cluster1 mirror image_14 started 2023-12-10T19:14:09.669 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:09.669 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:09.669 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_14 2023-12-10T19:14:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=started 2023-12-10T19:14:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_result 2023-12-10T19:14:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=stopped 2023-12-10T19:14:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_14 2023-12-10T19:14:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_instance=cluster1 2023-12-10T19:14:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:14:09.671 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:09.671 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance=cluster1 2023-12-10T19:14:09.671 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loop_instance 2023-12-10T19:14:09.671 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ + sleep 0 2023-12-10T19:14:09.671 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:grep -i state 2023-12-10T19:14:09.671 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T19:14:09.671 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T19:14:09.672 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_14 2023-12-10T19:14:09.673 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T19:14:09.673 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:14:09.673 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:14:09.673 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T19:14:09.673 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:14:09.673 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:14:09.673 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:14:09.673 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:14:09.673 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:14:09.673 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:14:09.673 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:14:09.673 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:14:09.673 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:14:09.673 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:14:09.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T19:14:09.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T19:14:09.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:09.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:14:09.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T19:14:09.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:14:09.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:14:09.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:14:09.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:14:09.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:14:09.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:14:09.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:14:09.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:14:09.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:14:09.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:14:09.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T19:14:09.699 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:14:09.699 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:14:09.699 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:14:09.699 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_14 2023-12-10T19:14:09.809 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2023-12-10T19:14:09.809 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:14:09.809 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:14:09.811 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_14 2023-12-10T19:14:09.811 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:1 2023-12-10T19:14:09.811 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:14:09.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:14:09.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-10T19:14:09.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:14:09.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:14:09.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:14:09.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:14:09.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:14:09.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:14:09.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:14:09.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:14:09.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:14:09.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:14:09.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-10T19:14:09.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:1 2023-12-10T19:14:09.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:09.813 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:14:09.813 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-10T19:14:09.813 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:14:09.813 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:14:09.813 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:14:09.813 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:14:09.813 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:14:09.813 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:14:09.813 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:14:09.813 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:14:09.814 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:14:09.814 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:14:09.814 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-10T19:14:09.834 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:14:09.834 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:14:09.835 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:14:09.835 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_14 2023-12-10T19:14:09.944 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2023-12-10T19:14:09.944 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:14:09.944 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:14:09.947 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_14 2023-12-10T19:14:09.947 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:2 2023-12-10T19:14:09.947 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:14:09.947 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:14:09.947 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-10T19:14:09.947 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:14:09.947 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:14:09.947 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:14:09.948 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:14:09.948 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:14:09.948 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:14:09.948 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:14:09.948 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:14:09.948 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:14:09.948 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:14:09.948 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-10T19:14:09.948 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:2 2023-12-10T19:14:09.948 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:09.948 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:14:09.948 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-10T19:14:09.948 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:14:09.948 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:14:09.948 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:14:09.948 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:14:09.949 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:14:09.949 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:14:09.949 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:14:09.949 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:14:09.949 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:14:09.949 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:14:09.949 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-10T19:14:09.971 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:14:09.971 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:14:09.971 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:14:09.971 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_14 2023-12-10T19:14:10.092 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:14:10.092 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ status_result= "state": "Replaying" 2023-12-10T19:14:10.093 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo "state": "Replaying" 2023-12-10T19:14:10.093 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep -i Replaying 2023-12-10T19:14:10.094 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "state": "Replaying" 2023-12-10T19:14:10.095 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=started 2023-12-10T19:14:10.095 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test started = started 2023-12-10T19:14:10.095 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:14:10.095 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_14 2023-12-10T19:14:10.095 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:14:10.095 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:14:10.095 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:10.096 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_14 2023-12-10T19:14:10.096 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = journal ] 2023-12-10T19:14:10.096 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = snapshot ] 2023-12-10T19:14:10.096 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_14 2023-12-10T19:14:10.096 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:14:10.096 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:14:10.096 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:10.096 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_14 2023-12-10T19:14:10.096 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster2-mirror-image_14.status 2023-12-10T19:14:10.096 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2-mirror-image_14.status 2023-12-10T19:14:10.096 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:14:10.097 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_14.status 2023-12-10T19:14:10.097 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-image_14.status 2023-12-10T19:14:10.097 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-image_14.status 2023-12-10T19:14:10.097 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:14:10.098 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_14.status 2023-12-10T19:14:10.099 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mirror_image_snapshot cluster2 mirror image_14 2023-12-10T19:14:10.099 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:14:10.099 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:10.099 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_14 2023-12-10T19:14:10.099 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_14 2023-12-10T19:14:10.693 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Snapshot ID: 145 2023-12-10T19:14:10.706 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_14 /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_14.status 2023-12-10T19:14:10.707 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:14:10.707 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:10.707 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_14 2023-12-10T19:14:10.707 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_14.status 2023-12-10T19:14:10.707 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_14 --format xml 2023-12-10T19:14:10.707 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:14:10.762 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-10T19:14:10.765 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snapshot_id=145 2023-12-10T19:14:10.765 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:14:10.765 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.2 2023-12-10T19:14:10.966 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_14.status 2023-12-10T19:14:10.966 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:10.966 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:10.966 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_14 2023-12-10T19:14:10.966 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_14.status 2023-12-10T19:14:10.967 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2023-12-10T19:14:10.967 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:14:11.034 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:14:11.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=81 2023-12-10T19:14:11.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 145 = 81 2023-12-10T19:14:11.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.4 2023-12-10T19:14:11.438 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_14.status 2023-12-10T19:14:11.438 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:11.439 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:11.439 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_14 2023-12-10T19:14:11.439 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_14.status 2023-12-10T19:14:11.439 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2023-12-10T19:14:11.439 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:14:11.507 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:14:11.510 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=81 2023-12-10T19:14:11.510 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 145 = 81 2023-12-10T19:14:11.510 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.8 2023-12-10T19:14:12.311 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_14.status 2023-12-10T19:14:12.312 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:12.312 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:12.312 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_14 2023-12-10T19:14:12.312 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_14.status 2023-12-10T19:14:12.312 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2023-12-10T19:14:12.312 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:14:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:14:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:14:12.381 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:14:12.381 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:14:12.381 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:14:12.381 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:14:12.381 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 145 = 2023-12-10T19:14:12.381 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1.6 2023-12-10T19:14:13.983 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_14.status 2023-12-10T19:14:13.983 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:13.983 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:13.983 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_14 2023-12-10T19:14:13.983 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_14.status 2023-12-10T19:14:13.983 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2023-12-10T19:14:13.983 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:14:14.051 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:14:14.054 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=145 2023-12-10T19:14:14.054 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 145 = 145 2023-12-10T19:14:14.054 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:14:14.054 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snap_present cluster1 mirror image_14 snap 2023-12-10T19:14:14.054 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:14.054 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:14.054 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_14 2023-12-10T19:14:14.054 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:14:14.054 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:14:14.054 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:14:15.055 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 info mirror/image_14@snap 2023-12-10T19:14:15.119 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_14': 2023-12-10T19:14:15.119 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:14:15.119 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:14:15.119 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 2 2023-12-10T19:14:15.119 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 10e8979ad9e9 2023-12-10T19:14:15.120 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.10e8979ad9e9 2023-12-10T19:14:15.120 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:14:15.120 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering, non-primary 2023-12-10T19:14:15.120 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:14:15.120 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:14:15.120 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:05:04 2023 2023-12-10T19:14:15.120 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:05:04 2023 2023-12-10T19:14:15.120 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:05:04 2023 2023-12-10T19:14:15.120 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: protected: False 2023-12-10T19:14:15.120 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:14:15.120 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:14:15.120 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: 1d54e641-de62-4511-9abb-01dd3218783c 2023-12-10T19:14:15.120 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: false 2023-12-10T19:14:15.123 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:14:15.123 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror image_14 snap 2023-12-10T19:14:15.123 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:15.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_14 2023-12-10T19:14:15.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:14:15.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:14:15.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_14.export 2023-12-10T19:14:15.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_14.export 2023-12-10T19:14:15.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_14.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_14.export 2023-12-10T19:14:15.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export image_14@snap /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_14.export 2023-12-10T19:14:15.329 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:14:15.336 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export image_14@snap /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_14.export 2023-12-10T19:14:15.590 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:14:15.599 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_14.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_14.export 2023-12-10T19:14:15.670 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_14.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_14.export 2023-12-10T19:14:15.710 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:14:15.710 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_15 2023-12-10T19:14:15.710 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror image_15 snap 2023-12-10T19:14:15.710 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:14:15.710 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:15.710 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_15 2023-12-10T19:14:15.710 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:14:15.710 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create image_15@snap --debug-rbd=20 --debug-journaler=20 2023-12-10T19:14:16.343 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_started cluster1 mirror image_15 2023-12-10T19:14:16.343 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:16.343 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:16.343 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_15 2023-12-10T19:14:16.344 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_state cluster1 mirror image_15 started 2023-12-10T19:14:16.344 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:16.344 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:16.344 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_15 2023-12-10T19:14:16.344 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=started 2023-12-10T19:14:16.344 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:14:16.344 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:14:17.344 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_replay_state cluster1 mirror image_15 started 2023-12-10T19:14:17.344 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:17.344 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:17.345 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_15 2023-12-10T19:14:17.345 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=started 2023-12-10T19:14:17.345 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_result 2023-12-10T19:14:17.345 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=stopped 2023-12-10T19:14:17.345 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_15 2023-12-10T19:14:17.345 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_instance=cluster1 2023-12-10T19:14:17.346 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:14:17.346 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ + local cluster=cluster1 2023-12-10T19:14:17.346 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance=cluster1 2023-12-10T19:14:17.346 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ localgrep loop_instance -i 2023-12-10T19:14:17.346 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: state 2023-12-10T19:14:17.346 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:14:17.346 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T19:14:17.347 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T19:14:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_15 2023-12-10T19:14:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T19:14:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:14:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:14:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T19:14:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:14:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:14:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:14:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:14:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:14:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:14:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:14:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:14:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:14:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:14:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T19:14:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T19:14:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:14:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T19:14:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:14:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:14:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:14:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:14:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:14:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:14:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:14:17.350 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:14:17.350 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:14:17.350 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:14:17.350 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T19:14:17.375 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:14:17.375 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:14:17.375 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:14:17.375 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_15 2023-12-10T19:14:17.498 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:14:17.499 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ status_result= "state": "Replaying" 2023-12-10T19:14:17.499 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo "state": "Replaying" 2023-12-10T19:14:17.499 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep -i Replaying 2023-12-10T19:14:17.500 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "state": "Replaying" 2023-12-10T19:14:17.501 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=started 2023-12-10T19:14:17.501 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test started = started 2023-12-10T19:14:17.501 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:14:17.501 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_15 2023-12-10T19:14:17.501 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:14:17.501 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:14:17.501 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:17.501 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_15 2023-12-10T19:14:17.501 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = journal ] 2023-12-10T19:14:17.501 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = snapshot ] 2023-12-10T19:14:17.501 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_15 2023-12-10T19:14:17.501 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:14:17.502 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:14:17.502 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:17.502 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_15 2023-12-10T19:14:17.502 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster2-mirror-image_15.status 2023-12-10T19:14:17.502 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2-mirror-image_15.status 2023-12-10T19:14:17.502 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:14:17.503 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_15.status 2023-12-10T19:14:17.503 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-image_15.status 2023-12-10T19:14:17.503 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-image_15.status 2023-12-10T19:14:17.503 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:14:17.503 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_15.status 2023-12-10T19:14:17.503 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mirror_image_snapshot cluster2 mirror image_15 2023-12-10T19:14:17.503 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:14:17.504 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:17.504 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_15 2023-12-10T19:14:17.504 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_15 2023-12-10T19:14:18.379 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Snapshot ID: 147 2023-12-10T19:14:18.392 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_15 /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_15.status 2023-12-10T19:14:18.392 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:14:18.392 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:18.392 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_15 2023-12-10T19:14:18.392 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_15.status 2023-12-10T19:14:18.392 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_15 --format xml 2023-12-10T19:14:18.393 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:14:18.448 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-10T19:14:18.452 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snapshot_id=147 2023-12-10T19:14:18.452 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:14:18.452 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.2 2023-12-10T19:14:18.654 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_15.status 2023-12-10T19:14:18.654 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:18.654 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:18.654 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_15 2023-12-10T19:14:18.654 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_15.status 2023-12-10T19:14:18.654 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2023-12-10T19:14:18.654 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:14:18.733 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:14:18.733 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:14:18.736 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:14:18.736 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:14:18.736 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:14:18.736 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:14:18.736 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 147 = 2023-12-10T19:14:18.736 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.4 2023-12-10T19:14:19.138 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_15.status 2023-12-10T19:14:19.138 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:19.138 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:19.138 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_15 2023-12-10T19:14:19.138 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_15.status 2023-12-10T19:14:19.138 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2023-12-10T19:14:19.138 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:14:19.211 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:14:19.212 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:14:19.213 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:14:19.214 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:14:19.214 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:14:19.214 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:14:19.214 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 147 = 2023-12-10T19:14:19.214 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.8 2023-12-10T19:14:20.015 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_15.status 2023-12-10T19:14:20.015 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:20.015 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:20.015 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_15 2023-12-10T19:14:20.015 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_15.status 2023-12-10T19:14:20.015 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2023-12-10T19:14:20.015 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:14:20.074 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:14:20.074 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:14:20.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:14:20.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:14:20.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:14:20.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:14:20.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 147 = 2023-12-10T19:14:20.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1.6 2023-12-10T19:14:21.678 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_15.status 2023-12-10T19:14:21.678 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:21.678 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:21.678 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_15 2023-12-10T19:14:21.678 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_15.status 2023-12-10T19:14:21.678 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2023-12-10T19:14:21.679 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:14:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:14:21.757 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=147 2023-12-10T19:14:21.758 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 147 = 147 2023-12-10T19:14:21.758 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:14:21.758 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snap_present cluster1 mirror image_15 snap 2023-12-10T19:14:21.758 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:21.758 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:21.758 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_15 2023-12-10T19:14:21.758 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:14:21.758 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:14:21.758 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:14:22.758 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 info mirror/image_15@snap 2023-12-10T19:14:22.847 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_15': 2023-12-10T19:14:22.847 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:14:22.848 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:14:22.848 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 2 2023-12-10T19:14:22.848 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 128bae02e2b9 2023-12-10T19:14:22.848 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.128bae02e2b9 2023-12-10T19:14:22.848 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:14:22.848 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering, non-primary 2023-12-10T19:14:22.848 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:14:22.848 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:14:22.848 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:05:15 2023 2023-12-10T19:14:22.848 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:05:15 2023 2023-12-10T19:14:22.848 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:05:15 2023 2023-12-10T19:14:22.848 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: protected: False 2023-12-10T19:14:22.848 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:14:22.848 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:14:22.848 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: e5868311-cdb8-441b-a90b-45641cca9efd 2023-12-10T19:14:22.849 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: false 2023-12-10T19:14:22.852 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:14:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror image_15 snap 2023-12-10T19:14:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_15 2023-12-10T19:14:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:14:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:14:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_15.export 2023-12-10T19:14:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_15.export 2023-12-10T19:14:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_15.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_15.export 2023-12-10T19:14:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export image_15@snap /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_15.export 2023-12-10T19:14:23.064 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:14:23.071 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export image_15@snap /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_15.export 2023-12-10T19:14:23.422 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:14:23.434 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_15.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_15.export 2023-12-10T19:14:23.511 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_15.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_15.export 2023-12-10T19:14:23.550 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:14:23.550 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_16 2023-12-10T19:14:23.550 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror image_16 snap 2023-12-10T19:14:23.550 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:14:23.550 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:23.550 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_16 2023-12-10T19:14:23.550 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:14:23.550 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create image_16@snap --debug-rbd=20 --debug-journaler=20 2023-12-10T19:14:23.683 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_started cluster1 mirror image_16 2023-12-10T19:14:23.683 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:23.683 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_16 2023-12-10T19:14:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_state cluster1 mirror image_16 started 2023-12-10T19:14:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_16 2023-12-10T19:14:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=started 2023-12-10T19:14:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:14:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:14:24.684 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_replay_state cluster1 mirror image_16 started 2023-12-10T19:14:24.684 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:24.685 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:24.685 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_16 2023-12-10T19:14:24.685 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=started 2023-12-10T19:14:24.685 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_result 2023-12-10T19:14:24.685 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=stopped 2023-12-10T19:14:24.685 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_16 2023-12-10T19:14:24.685 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_instance=cluster1 2023-12-10T19:14:24.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:14:24.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ + local cluster=cluster1 2023-12-10T19:14:24.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance=cluster1 2023-12-10T19:14:24.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loop_instance 2023-12-10T19:14:24.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grepsleep -i 0 state 2023-12-10T19:14:24.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:14:24.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T19:14:24.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T19:14:24.688 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_16 2023-12-10T19:14:24.688 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T19:14:24.688 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:14:24.689 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:14:24.689 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T19:14:24.689 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:14:24.689 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:14:24.689 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:14:24.689 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:14:24.689 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:14:24.689 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:14:24.689 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:14:24.689 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:14:24.689 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:14:24.689 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:14:24.689 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T19:14:24.689 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T19:14:24.689 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:24.689 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:14:24.689 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T19:14:24.690 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:14:24.690 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:14:24.690 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:14:24.690 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:14:24.690 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:14:24.690 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:14:24.690 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:14:24.690 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:14:24.690 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:14:24.690 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:14:24.690 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T19:14:24.717 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:14:24.717 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:14:24.717 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:14:24.717 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_16 2023-12-10T19:14:24.837 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2023-12-10T19:14:24.837 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:14:24.837 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:14:24.838 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_16 2023-12-10T19:14:24.839 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:1 2023-12-10T19:14:24.839 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:14:24.839 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:14:24.839 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-10T19:14:24.839 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:14:24.839 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:14:24.839 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:14:24.839 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:14:24.839 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:14:24.839 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:14:24.839 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:14:24.839 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:14:24.839 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:14:24.839 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:14:24.840 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-10T19:14:24.840 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:1 2023-12-10T19:14:24.840 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:24.841 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:14:24.841 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-10T19:14:24.841 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:14:24.841 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:14:24.841 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:14:24.841 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:14:24.841 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:14:24.841 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:14:24.841 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:14:24.841 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:14:24.841 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:14:24.841 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:14:24.841 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-10T19:14:24.858 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:14:24.859 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:14:24.859 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:14:24.859 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_16 2023-12-10T19:14:24.975 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2023-12-10T19:14:24.975 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:14:24.975 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:14:24.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_16 2023-12-10T19:14:24.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:2 2023-12-10T19:14:24.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:14:24.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:14:24.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-10T19:14:24.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:14:24.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:14:24.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:14:24.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:14:24.978 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:14:24.978 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:14:24.978 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:14:24.978 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:14:24.978 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:14:24.978 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:14:24.978 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-10T19:14:24.978 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:2 2023-12-10T19:14:24.978 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:24.978 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:14:24.978 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-10T19:14:24.978 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:14:24.978 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:14:24.978 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:14:24.978 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:14:24.978 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:14:24.979 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:14:24.979 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:14:24.979 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:14:24.979 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:14:24.979 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:14:24.979 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-10T19:14:25.001 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:14:25.002 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:14:25.002 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:14:25.002 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_16 2023-12-10T19:14:25.129 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2023-12-10T19:14:25.129 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:14:25.130 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:14:25.134 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_16 2023-12-10T19:14:25.134 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:3 2023-12-10T19:14:25.134 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:14:25.134 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:14:25.134 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-10T19:14:25.134 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:3 2023-12-10T19:14:25.134 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:14:25.134 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:14:25.134 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:14:25.135 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:14:25.135 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 3 = cluster1:3 ] 2023-12-10T19:14:25.135 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:14:25.135 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:14:25.135 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=3 2023-12-10T19:14:25.135 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:14:25.135 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-10T19:14:25.135 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:3 2023-12-10T19:14:25.135 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:25.135 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:14:25.135 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-10T19:14:25.135 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:3 2023-12-10T19:14:25.135 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:14:25.135 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:14:25.135 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:14:25.135 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:14:25.136 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 3 = cluster1:3 ] 2023-12-10T19:14:25.136 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:14:25.136 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:14:25.136 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=3 2023-12-10T19:14:25.136 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:14:25.136 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-10T19:14:25.156 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T19:14:25.156 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T19:14:25.156 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T19:14:25.156 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_16 2023-12-10T19:14:25.285 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:14:25.286 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ status_result= "state": "Replaying" 2023-12-10T19:14:25.286 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo "state": "Replaying" 2023-12-10T19:14:25.286 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep -i Replaying 2023-12-10T19:14:25.287 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "state": "Replaying" 2023-12-10T19:14:25.288 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=started 2023-12-10T19:14:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test started = started 2023-12-10T19:14:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:14:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_16 2023-12-10T19:14:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:14:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:14:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_16 2023-12-10T19:14:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = journal ] 2023-12-10T19:14:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = snapshot ] 2023-12-10T19:14:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_16 2023-12-10T19:14:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:14:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:14:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_16 2023-12-10T19:14:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster2-mirror-image_16.status 2023-12-10T19:14:25.290 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2-mirror-image_16.status 2023-12-10T19:14:25.290 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:14:25.290 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_16.status 2023-12-10T19:14:25.290 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-image_16.status 2023-12-10T19:14:25.291 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-image_16.status 2023-12-10T19:14:25.291 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:14:25.291 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_16.status 2023-12-10T19:14:25.291 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mirror_image_snapshot cluster2 mirror image_16 2023-12-10T19:14:25.291 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:14:25.291 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:25.291 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_16 2023-12-10T19:14:25.291 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_16 2023-12-10T19:14:26.357 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Snapshot ID: 149 2023-12-10T19:14:26.371 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_16 /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_16.status 2023-12-10T19:14:26.371 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:14:26.371 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:26.371 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_16 2023-12-10T19:14:26.371 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_16.status 2023-12-10T19:14:26.371 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_16 --format xml 2023-12-10T19:14:26.371 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:14:26.426 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-10T19:14:26.428 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snapshot_id=149 2023-12-10T19:14:26.428 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:14:26.428 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.2 2023-12-10T19:14:26.630 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_16.status 2023-12-10T19:14:26.630 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:26.630 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:26.630 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_16 2023-12-10T19:14:26.630 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_16.status 2023-12-10T19:14:26.630 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2023-12-10T19:14:26.630 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:14:26.729 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:14:26.729 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:14:26.731 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:14:26.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:14:26.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:14:26.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:14:26.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 149 = 2023-12-10T19:14:26.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.4 2023-12-10T19:14:26.864 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5160287408105054) > weight (0.25) 2023-12-10T19:14:26.864 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.7089309745677921) > weight (0.25) 2023-12-10T19:14:26.864 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.2785725635691191) > weight (0.25) 2023-12-10T19:14:26.864 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.9349130491446577) > weight (0.25) 2023-12-10T19:14:26.864 INFO:tasks.rbd_mirror_thrash:waiting for 77 secs before thrashing 2023-12-10T19:14:27.133 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_16.status 2023-12-10T19:14:27.133 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:27.133 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:27.133 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_16 2023-12-10T19:14:27.133 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_16.status 2023-12-10T19:14:27.133 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2023-12-10T19:14:27.134 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:14:27.207 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:14:27.208 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:14:27.209 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:14:27.209 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:14:27.209 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:14:27.210 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:14:27.210 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 149 = 2023-12-10T19:14:27.210 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.8 2023-12-10T19:14:28.011 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_16.status 2023-12-10T19:14:28.011 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:28.011 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:28.011 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_16 2023-12-10T19:14:28.012 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_16.status 2023-12-10T19:14:28.012 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2023-12-10T19:14:28.012 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:14:28.082 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:14:28.084 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=149 2023-12-10T19:14:28.084 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 149 = 149 2023-12-10T19:14:28.084 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:14:28.085 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snap_present cluster1 mirror image_16 snap 2023-12-10T19:14:28.085 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:28.085 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:28.085 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_16 2023-12-10T19:14:28.085 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:14:28.085 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:14:28.085 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:14:29.085 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 info mirror/image_16@snap 2023-12-10T19:14:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_16': 2023-12-10T19:14:29.147 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:14:29.147 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:14:29.147 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 2 2023-12-10T19:14:29.147 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 128b13d2e210 2023-12-10T19:14:29.147 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.128b13d2e210 2023-12-10T19:14:29.147 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:14:29.147 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering, non-primary 2023-12-10T19:14:29.147 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:14:29.147 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:14:29.147 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:05:22 2023 2023-12-10T19:14:29.147 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:05:22 2023 2023-12-10T19:14:29.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:05:22 2023 2023-12-10T19:14:29.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: protected: False 2023-12-10T19:14:29.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:14:29.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:14:29.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: 66ef9828-72b1-4f4f-b6ef-90d776f972ce 2023-12-10T19:14:29.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: false 2023-12-10T19:14:29.151 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:14:29.151 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror image_16 snap 2023-12-10T19:14:29.151 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:29.151 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_16 2023-12-10T19:14:29.152 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:14:29.152 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:14:29.152 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_16.export 2023-12-10T19:14:29.152 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_16.export 2023-12-10T19:14:29.152 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_16.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_16.export 2023-12-10T19:14:29.152 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export image_16@snap /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_16.export 2023-12-10T19:14:29.362 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:14:29.370 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export image_16@snap /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_16.export 2023-12-10T19:14:29.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:14:29.688 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_16.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_16.export 2023-12-10T19:14:29.766 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_16.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_16.export 2023-12-10T19:14:29.805 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:14:29.805 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_17 2023-12-10T19:14:29.805 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror image_17 snap 2023-12-10T19:14:29.805 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:14:29.805 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:29.805 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_17 2023-12-10T19:14:29.805 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:14:29.805 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create image_17@snap --debug-rbd=20 --debug-journaler=20 2023-12-10T19:14:30.684 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_started cluster1 mirror image_17 2023-12-10T19:14:30.684 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:30.684 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:30.684 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_17 2023-12-10T19:14:30.684 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_state cluster1 mirror image_17 started 2023-12-10T19:14:30.684 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:30.684 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:30.684 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_17 2023-12-10T19:14:30.684 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=started 2023-12-10T19:14:30.684 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:14:30.685 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:14:31.685 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_replay_state cluster1 mirror image_17 started 2023-12-10T19:14:31.685 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:31.685 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:31.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_17 2023-12-10T19:14:31.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=started 2023-12-10T19:14:31.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_result 2023-12-10T19:14:31.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=stopped 2023-12-10T19:14:31.690 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_17 2023-12-10T19:14:31.690 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_instance=cluster1 2023-12-10T19:14:31.690 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift+ 2023-12-10T19:14:31.690 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:31.690 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:grep -i state 2023-12-10T19:14:31.690 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance=cluster1 2023-12-10T19:14:31.691 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loop_instance 2023-12-10T19:14:31.691 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:14:31.691 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T19:14:31.691 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T19:14:31.691 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_17 2023-12-10T19:14:31.691 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T19:14:31.691 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:14:31.692 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:14:31.692 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T19:14:31.692 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:14:31.692 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:14:31.692 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:14:31.692 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:14:31.692 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:14:31.693 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:14:31.693 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:14:31.693 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:14:31.693 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:14:31.693 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:14:31.693 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T19:14:31.693 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T19:14:31.693 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:31.694 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:14:31.694 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T19:14:31.694 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:14:31.694 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:14:31.694 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:14:31.694 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:14:31.695 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:14:31.695 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:14:31.695 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:14:31.695 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:14:31.695 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:14:31.695 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:14:31.695 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T19:14:31.711 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:14:31.711 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:14:31.711 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:14:31.711 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_17 2023-12-10T19:14:31.820 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2023-12-10T19:14:31.820 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:14:31.820 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:14:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_17 2023-12-10T19:14:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:1 2023-12-10T19:14:31.824 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:14:31.824 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:14:31.824 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-10T19:14:31.824 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:14:31.824 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:14:31.824 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:14:31.824 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:14:31.824 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:14:31.824 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:14:31.824 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:14:31.824 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:14:31.824 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:14:31.824 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:14:31.824 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-10T19:14:31.824 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:1 2023-12-10T19:14:31.824 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:31.825 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:14:31.825 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-10T19:14:31.825 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:14:31.825 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:14:31.825 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:14:31.825 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:14:31.825 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:14:31.825 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:14:31.825 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:14:31.825 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:14:31.825 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:14:31.825 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:14:31.825 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-10T19:14:31.846 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:14:31.847 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:14:31.847 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:14:31.847 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_17 2023-12-10T19:14:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2023-12-10T19:14:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:14:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:14:31.963 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_17 2023-12-10T19:14:31.963 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:2 2023-12-10T19:14:31.963 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:14:31.963 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:14:31.963 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-10T19:14:31.963 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:14:31.963 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:14:31.963 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:14:31.964 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:14:31.964 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:14:31.964 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:14:31.964 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:14:31.964 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:14:31.964 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:14:31.964 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:14:31.964 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-10T19:14:31.964 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:2 2023-12-10T19:14:31.964 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:31.964 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:14:31.964 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-10T19:14:31.964 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:14:31.964 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:14:31.964 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:14:31.965 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:14:31.965 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:14:31.965 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:14:31.965 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:14:31.965 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:14:31.965 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:14:31.965 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:14:31.965 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-10T19:14:31.983 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:14:31.983 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:14:31.983 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:14:31.983 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_17 2023-12-10T19:14:32.111 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2023-12-10T19:14:32.111 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:14:32.111 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:14:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_17 2023-12-10T19:14:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:3 2023-12-10T19:14:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:14:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:14:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-10T19:14:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:3 2023-12-10T19:14:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:14:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:14:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:14:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:14:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 3 = cluster1:3 ] 2023-12-10T19:14:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:14:32.116 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:14:32.116 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=3 2023-12-10T19:14:32.116 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:14:32.116 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-10T19:14:32.116 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:3 2023-12-10T19:14:32.116 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:32.116 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:14:32.116 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-10T19:14:32.116 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:3 2023-12-10T19:14:32.116 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:14:32.116 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:14:32.116 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:14:32.116 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:14:32.116 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 3 = cluster1:3 ] 2023-12-10T19:14:32.116 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:14:32.116 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:14:32.116 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=3 2023-12-10T19:14:32.117 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:14:32.117 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-10T19:14:32.137 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T19:14:32.137 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T19:14:32.137 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T19:14:32.137 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_17 2023-12-10T19:14:32.270 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:14:32.271 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ status_result= "state": "Replaying" 2023-12-10T19:14:32.271 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo+ "state": "Replaying" 2023-12-10T19:14:32.271 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:grep -i Replaying 2023-12-10T19:14:32.272 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "state": "Replaying" 2023-12-10T19:14:32.274 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=started 2023-12-10T19:14:32.274 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test started = started 2023-12-10T19:14:32.274 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:14:32.274 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_17 2023-12-10T19:14:32.274 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:14:32.274 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:14:32.274 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:32.274 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_17 2023-12-10T19:14:32.274 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = journal ] 2023-12-10T19:14:32.274 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = snapshot ] 2023-12-10T19:14:32.275 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_17 2023-12-10T19:14:32.275 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:14:32.275 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:14:32.275 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:32.275 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_17 2023-12-10T19:14:32.275 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster2-mirror-image_17.status 2023-12-10T19:14:32.275 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2-mirror-image_17.status 2023-12-10T19:14:32.275 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:14:32.276 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_17.status 2023-12-10T19:14:32.276 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-image_17.status 2023-12-10T19:14:32.276 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-image_17.status 2023-12-10T19:14:32.276 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:14:32.276 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_17.status 2023-12-10T19:14:32.276 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mirror_image_snapshot cluster2 mirror image_17 2023-12-10T19:14:32.276 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:14:32.276 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:32.276 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_17 2023-12-10T19:14:32.277 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_17 2023-12-10T19:14:32.710 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Snapshot ID: 151 2023-12-10T19:14:32.724 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_17 /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_17.status 2023-12-10T19:14:32.724 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:14:32.724 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:32.724 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_17 2023-12-10T19:14:32.724 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_17.status 2023-12-10T19:14:32.724 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_17 --format xml 2023-12-10T19:14:32.725 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:14:32.781 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-10T19:14:32.783 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snapshot_id=151 2023-12-10T19:14:32.783 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:14:32.783 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.2 2023-12-10T19:14:32.984 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_17.status 2023-12-10T19:14:32.984 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:32.984 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:32.984 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_17 2023-12-10T19:14:32.984 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_17.status 2023-12-10T19:14:32.985 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2023-12-10T19:14:32.985 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:14:33.057 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:14:33.060 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=84 2023-12-10T19:14:33.060 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 151 = 84 2023-12-10T19:14:33.060 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.4 2023-12-10T19:14:33.461 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_17.status 2023-12-10T19:14:33.461 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:33.461 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:33.461 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_17 2023-12-10T19:14:33.461 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_17.status 2023-12-10T19:14:33.461 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2023-12-10T19:14:33.461 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:14:33.531 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:14:33.534 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=84 2023-12-10T19:14:33.534 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 151 = 84 2023-12-10T19:14:33.534 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.8 2023-12-10T19:14:34.335 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_17.status 2023-12-10T19:14:34.335 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:34.335 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:34.336 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_17 2023-12-10T19:14:34.336 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_17.status 2023-12-10T19:14:34.336 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2023-12-10T19:14:34.336 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:14:34.406 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:14:34.406 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:14:34.408 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:14:34.408 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:14:34.408 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:14:34.409 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:14:34.409 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 151 = 2023-12-10T19:14:34.409 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1.6 2023-12-10T19:14:36.010 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_17.status 2023-12-10T19:14:36.010 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:36.010 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:36.010 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_17 2023-12-10T19:14:36.011 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_17.status 2023-12-10T19:14:36.011 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2023-12-10T19:14:36.011 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:14:36.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:14:36.078 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=151 2023-12-10T19:14:36.078 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 151 = 151 2023-12-10T19:14:36.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:14:36.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snap_present cluster1 mirror image_17 snap 2023-12-10T19:14:36.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:36.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:36.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_17 2023-12-10T19:14:36.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:14:36.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:14:36.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:14:37.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 info mirror/image_17@snap 2023-12-10T19:14:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_17': 2023-12-10T19:14:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:14:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:14:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 2 2023-12-10T19:14:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 128bf94e06b4 2023-12-10T19:14:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.128bf94e06b4 2023-12-10T19:14:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:14:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering, non-primary 2023-12-10T19:14:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:14:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:14:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:05:32 2023 2023-12-10T19:14:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:05:32 2023 2023-12-10T19:14:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:05:32 2023 2023-12-10T19:14:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: protected: False 2023-12-10T19:14:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:14:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:14:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: a44b2376-7333-4134-9c68-c71acef2b46e 2023-12-10T19:14:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: false 2023-12-10T19:14:37.143 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:14:37.143 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror image_17 snap 2023-12-10T19:14:37.144 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:37.144 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_17 2023-12-10T19:14:37.144 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:14:37.144 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:14:37.144 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_17.export 2023-12-10T19:14:37.144 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_17.export 2023-12-10T19:14:37.144 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_17.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_17.export 2023-12-10T19:14:37.144 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export image_17@snap /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_17.export 2023-12-10T19:14:37.370 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:14:37.382 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export image_17@snap /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_17.export 2023-12-10T19:14:37.705 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:14:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_17.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_17.export 2023-12-10T19:14:37.798 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_17.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_17.export 2023-12-10T19:14:37.838 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:14:37.838 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_18 2023-12-10T19:14:37.838 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror image_18 snap 2023-12-10T19:14:37.838 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:14:37.838 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:37.838 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_18 2023-12-10T19:14:37.838 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:14:37.838 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create image_18@snap --debug-rbd=20 --debug-journaler=20 2023-12-10T19:14:38.688 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_started cluster1 mirror image_18 2023-12-10T19:14:38.688 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:38.689 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:38.689 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_18 2023-12-10T19:14:38.689 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_state cluster1 mirror image_18 started 2023-12-10T19:14:38.689 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:38.689 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:38.689 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_18 2023-12-10T19:14:38.689 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=started 2023-12-10T19:14:38.689 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:14:38.689 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:14:39.689 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_replay_state cluster1 mirror image_18 started 2023-12-10T19:14:39.690 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:39.690 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:39.690 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_18 2023-12-10T19:14:39.690 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=started 2023-12-10T19:14:39.690 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_result 2023-12-10T19:14:39.690 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=stopped 2023-12-10T19:14:39.691 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_18 2023-12-10T19:14:39.691 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ + local cluster_instance=cluster1 2023-12-10T19:14:39.691 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:14:39.691 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:grep+ -ilocal state cluster=cluster1 2023-12-10T19:14:39.691 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:14:39.691 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance=cluster1 2023-12-10T19:14:39.691 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loop_instance 2023-12-10T19:14:39.691 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:14:39.692 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T19:14:39.692 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T19:14:39.694 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_18 2023-12-10T19:14:39.694 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T19:14:39.694 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:14:39.694 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:14:39.694 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T19:14:39.694 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:14:39.694 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:14:39.694 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:14:39.694 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:14:39.694 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:14:39.694 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:14:39.694 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:14:39.694 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:14:39.695 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:14:39.695 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:14:39.695 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T19:14:39.695 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T19:14:39.695 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:39.695 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:14:39.695 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T19:14:39.695 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:14:39.695 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:14:39.695 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:14:39.695 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:14:39.695 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:14:39.695 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:14:39.696 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:14:39.696 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:14:39.696 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:14:39.696 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:14:39.696 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T19:14:39.714 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:14:39.715 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:14:39.715 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:14:39.715 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_18 2023-12-10T19:14:39.831 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2023-12-10T19:14:39.831 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:14:39.831 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:14:39.834 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_18 2023-12-10T19:14:39.834 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:1 2023-12-10T19:14:39.834 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:14:39.834 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:14:39.834 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-10T19:14:39.834 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:14:39.834 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:14:39.834 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:14:39.834 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:14:39.835 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:14:39.835 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:14:39.835 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:14:39.835 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:14:39.835 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:14:39.835 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:14:39.835 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-10T19:14:39.835 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:1 2023-12-10T19:14:39.835 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:39.835 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:14:39.835 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-10T19:14:39.836 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:14:39.836 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:14:39.836 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:14:39.836 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:14:39.836 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:14:39.836 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:14:39.836 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:14:39.836 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:14:39.836 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:14:39.836 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:14:39.836 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-10T19:14:39.856 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:14:39.856 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:14:39.856 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:14:39.856 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_18 2023-12-10T19:14:39.971 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2023-12-10T19:14:39.971 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:14:39.971 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:14:39.973 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_18 2023-12-10T19:14:39.974 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:2 2023-12-10T19:14:39.974 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:14:39.974 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:14:39.974 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-10T19:14:39.974 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:14:39.974 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:14:39.974 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:14:39.974 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:14:39.974 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:14:39.974 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:14:39.974 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:14:39.974 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:14:39.974 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:14:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:14:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-10T19:14:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:2 2023-12-10T19:14:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:14:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-10T19:14:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:14:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:14:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:14:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:14:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:14:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:14:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:14:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:14:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:14:39.976 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:14:39.976 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-10T19:14:39.997 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:14:39.997 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:14:39.998 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:14:39.998 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_18 2023-12-10T19:14:40.110 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2023-12-10T19:14:40.111 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:14:40.111 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:14:40.115 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_18 2023-12-10T19:14:40.115 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:3 2023-12-10T19:14:40.115 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:14:40.115 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:14:40.115 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-10T19:14:40.115 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:3 2023-12-10T19:14:40.115 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:14:40.115 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:14:40.115 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:14:40.115 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:14:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 3 = cluster1:3 ] 2023-12-10T19:14:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:14:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:14:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=3 2023-12-10T19:14:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:14:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-10T19:14:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:3 2023-12-10T19:14:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:14:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-10T19:14:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:3 2023-12-10T19:14:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:14:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:14:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:14:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:14:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 3 = cluster1:3 ] 2023-12-10T19:14:40.117 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:14:40.117 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:14:40.117 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=3 2023-12-10T19:14:40.117 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:14:40.117 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-10T19:14:40.137 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T19:14:40.137 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T19:14:40.137 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T19:14:40.137 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_18 2023-12-10T19:14:40.271 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:14:40.271 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ status_result= "state": "Replaying" 2023-12-10T19:14:40.271 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo "state": "Replaying" 2023-12-10T19:14:40.271 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep -i Replaying 2023-12-10T19:14:40.273 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "state": "Replaying" 2023-12-10T19:14:40.274 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=started 2023-12-10T19:14:40.274 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test started = started 2023-12-10T19:14:40.274 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:14:40.274 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_18 2023-12-10T19:14:40.275 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:14:40.275 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:14:40.275 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:40.275 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_18 2023-12-10T19:14:40.275 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = journal ] 2023-12-10T19:14:40.275 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = snapshot ] 2023-12-10T19:14:40.275 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_18 2023-12-10T19:14:40.275 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:14:40.275 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:14:40.275 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:40.275 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_18 2023-12-10T19:14:40.275 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster2-mirror-image_18.status 2023-12-10T19:14:40.275 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2-mirror-image_18.status 2023-12-10T19:14:40.275 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:14:40.276 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_18.status 2023-12-10T19:14:40.276 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-image_18.status 2023-12-10T19:14:40.276 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-image_18.status 2023-12-10T19:14:40.276 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:14:40.277 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_18.status 2023-12-10T19:14:40.277 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mirror_image_snapshot cluster2 mirror image_18 2023-12-10T19:14:40.277 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:14:40.277 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:40.277 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_18 2023-12-10T19:14:40.277 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_18 2023-12-10T19:14:40.694 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Snapshot ID: 153 2023-12-10T19:14:40.706 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_18 /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_18.status 2023-12-10T19:14:40.706 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:14:40.706 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:40.706 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_18 2023-12-10T19:14:40.706 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_18.status 2023-12-10T19:14:40.706 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_18 --format xml 2023-12-10T19:14:40.707 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:14:40.763 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-10T19:14:40.766 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snapshot_id=153 2023-12-10T19:14:40.766 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:14:40.766 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.2 2023-12-10T19:14:40.967 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_18.status 2023-12-10T19:14:40.967 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:40.967 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:40.967 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_18 2023-12-10T19:14:40.967 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_18.status 2023-12-10T19:14:40.968 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2023-12-10T19:14:40.968 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:14:41.033 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:14:41.035 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=85 2023-12-10T19:14:41.035 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 153 = 85 2023-12-10T19:14:41.035 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.4 2023-12-10T19:14:41.437 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_18.status 2023-12-10T19:14:41.437 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:41.437 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:41.437 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_18 2023-12-10T19:14:41.437 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_18.status 2023-12-10T19:14:41.437 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2023-12-10T19:14:41.438 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:14:41.505 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:14:41.508 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=85 2023-12-10T19:14:41.508 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 153 = 85 2023-12-10T19:14:41.508 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.8 2023-12-10T19:14:42.309 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_18.status 2023-12-10T19:14:42.309 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:42.309 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:42.309 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_18 2023-12-10T19:14:42.309 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_18.status 2023-12-10T19:14:42.309 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2023-12-10T19:14:42.310 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:14:42.377 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:14:42.377 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:14:42.379 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:14:42.380 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:14:42.380 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:14:42.380 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:14:42.380 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 153 = 2023-12-10T19:14:42.380 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1.6 2023-12-10T19:14:43.982 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_18.status 2023-12-10T19:14:43.982 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:43.982 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:43.982 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_18 2023-12-10T19:14:43.982 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_18.status 2023-12-10T19:14:43.982 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2023-12-10T19:14:43.982 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:14:44.049 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:14:44.052 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=153 2023-12-10T19:14:44.052 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 153 = 153 2023-12-10T19:14:44.052 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:14:44.052 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snap_present cluster1 mirror image_18 snap 2023-12-10T19:14:44.052 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:44.052 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:44.052 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_18 2023-12-10T19:14:44.053 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:14:44.053 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:14:44.053 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:14:45.054 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 info mirror/image_18@snap 2023-12-10T19:14:45.117 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_18': 2023-12-10T19:14:45.117 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:14:45.117 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:14:45.117 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 2 2023-12-10T19:14:45.117 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 10e8c2d51758 2023-12-10T19:14:45.117 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.10e8c2d51758 2023-12-10T19:14:45.118 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:14:45.118 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering, non-primary 2023-12-10T19:14:45.118 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:14:45.118 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:14:45.118 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:05:41 2023 2023-12-10T19:14:45.118 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:05:41 2023 2023-12-10T19:14:45.118 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:05:41 2023 2023-12-10T19:14:45.118 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: protected: False 2023-12-10T19:14:45.118 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:14:45.118 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:14:45.118 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: 013b5e35-24cb-486f-93bf-0f941ebb1c1d 2023-12-10T19:14:45.118 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: false 2023-12-10T19:14:45.122 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:14:45.122 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror image_18 snap 2023-12-10T19:14:45.123 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:45.123 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_18 2023-12-10T19:14:45.123 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:14:45.123 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:14:45.123 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_18.export 2023-12-10T19:14:45.123 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_18.export 2023-12-10T19:14:45.123 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_18.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_18.export 2023-12-10T19:14:45.123 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export image_18@snap /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_18.export 2023-12-10T19:14:45.344 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:14:45.355 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export image_18@snap /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_18.export 2023-12-10T19:14:45.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:14:45.743 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_18.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_18.export 2023-12-10T19:14:45.821 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_18.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_18.export 2023-12-10T19:14:45.861 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:14:45.861 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_19 2023-12-10T19:14:45.861 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror image_19 snap 2023-12-10T19:14:45.861 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:14:45.861 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:45.861 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_19 2023-12-10T19:14:45.861 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:14:45.861 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create image_19@snap --debug-rbd=20 --debug-journaler=20 2023-12-10T19:14:46.351 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_started cluster1 mirror image_19 2023-12-10T19:14:46.351 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:46.351 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:46.351 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_19 2023-12-10T19:14:46.351 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_state cluster1 mirror image_19 started 2023-12-10T19:14:46.351 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:46.351 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:46.352 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_19 2023-12-10T19:14:46.352 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=started 2023-12-10T19:14:46.352 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:14:46.352 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:14:47.352 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_replay_state cluster1 mirror image_19 started 2023-12-10T19:14:47.353 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:47.353 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:47.353 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_19 2023-12-10T19:14:47.353 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=started 2023-12-10T19:14:47.353 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_result 2023-12-10T19:14:47.353 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=stopped 2023-12-10T19:14:47.354 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_19 2023-12-10T19:14:47.354 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_instance=cluster1 2023-12-10T19:14:47.354 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:14:47.354 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:47.354 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance=cluster1 2023-12-10T19:14:47.354 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loop_instance 2023-12-10T19:14:47.354 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ + sleep 0 2023-12-10T19:14:47.354 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:grep -i state 2023-12-10T19:14:47.355 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T19:14:47.355 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T19:14:47.356 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_19 2023-12-10T19:14:47.356 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T19:14:47.356 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:14:47.356 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:14:47.356 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T19:14:47.356 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:14:47.356 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:14:47.356 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:14:47.356 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:14:47.356 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:14:47.356 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:14:47.357 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:14:47.357 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:14:47.357 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:14:47.357 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:14:47.357 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T19:14:47.357 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T19:14:47.357 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:47.357 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:14:47.357 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T19:14:47.357 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:14:47.357 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:14:47.357 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:14:47.357 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:14:47.357 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:14:47.357 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:14:47.358 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:14:47.358 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:14:47.358 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:14:47.358 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:14:47.358 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T19:14:47.381 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:14:47.382 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:14:47.382 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:14:47.382 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_19 2023-12-10T19:14:47.510 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:14:47.511 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ status_result= "state": "Replaying" 2023-12-10T19:14:47.511 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo "state": "Replaying" 2023-12-10T19:14:47.511 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep -i Replaying 2023-12-10T19:14:47.512 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "state": "Replaying" 2023-12-10T19:14:47.513 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=started 2023-12-10T19:14:47.513 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test started = started 2023-12-10T19:14:47.513 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:14:47.514 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_19 2023-12-10T19:14:47.514 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:14:47.514 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:14:47.514 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:47.514 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_19 2023-12-10T19:14:47.514 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = journal ] 2023-12-10T19:14:47.514 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = snapshot ] 2023-12-10T19:14:47.514 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_19 2023-12-10T19:14:47.514 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:14:47.514 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:14:47.514 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:47.514 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_19 2023-12-10T19:14:47.515 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster2-mirror-image_19.status 2023-12-10T19:14:47.516 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2-mirror-image_19.status 2023-12-10T19:14:47.516 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:14:47.516 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_19.status 2023-12-10T19:14:47.516 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-image_19.status 2023-12-10T19:14:47.516 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-image_19.status 2023-12-10T19:14:47.516 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:14:47.516 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_19.status 2023-12-10T19:14:47.516 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mirror_image_snapshot cluster2 mirror image_19 2023-12-10T19:14:47.517 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:14:47.517 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:47.517 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_19 2023-12-10T19:14:47.517 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_19 2023-12-10T19:14:48.378 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Snapshot ID: 155 2023-12-10T19:14:48.393 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_19 /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_19.status 2023-12-10T19:14:48.393 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:14:48.394 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:48.394 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_19 2023-12-10T19:14:48.394 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_19.status 2023-12-10T19:14:48.394 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_19 --format xml 2023-12-10T19:14:48.394 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:14:48.449 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-10T19:14:48.452 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snapshot_id=155 2023-12-10T19:14:48.452 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:14:48.452 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.2 2023-12-10T19:14:48.653 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_19.status 2023-12-10T19:14:48.654 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:48.654 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:48.654 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_19 2023-12-10T19:14:48.654 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_19.status 2023-12-10T19:14:48.654 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2023-12-10T19:14:48.654 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:14:48.721 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:14:48.721 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:14:48.723 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:14:48.723 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:14:48.723 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:14:48.723 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:14:48.724 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 155 = 2023-12-10T19:14:48.724 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.4 2023-12-10T19:14:49.125 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_19.status 2023-12-10T19:14:49.125 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:49.125 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:49.125 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_19 2023-12-10T19:14:49.125 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_19.status 2023-12-10T19:14:49.125 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2023-12-10T19:14:49.125 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:14:49.199 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:14:49.199 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:14:49.201 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:14:49.201 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:14:49.201 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:14:49.202 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:14:49.202 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 155 = 2023-12-10T19:14:49.202 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.8 2023-12-10T19:14:50.003 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_19.status 2023-12-10T19:14:50.003 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:50.003 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:50.004 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_19 2023-12-10T19:14:50.004 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_19.status 2023-12-10T19:14:50.004 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2023-12-10T19:14:50.004 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:14:50.070 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:14:50.072 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=155 2023-12-10T19:14:50.072 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 155 = 155 2023-12-10T19:14:50.072 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:14:50.073 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snap_present cluster1 mirror image_19 snap 2023-12-10T19:14:50.073 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:50.073 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:50.073 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_19 2023-12-10T19:14:50.073 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:14:50.073 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:14:50.073 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:14:51.073 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 info mirror/image_19@snap 2023-12-10T19:14:51.136 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_19': 2023-12-10T19:14:51.136 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:14:51.136 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:14:51.136 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 2 2023-12-10T19:14:51.136 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 1200afe9af7d 2023-12-10T19:14:51.136 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.1200afe9af7d 2023-12-10T19:14:51.136 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:14:51.136 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering, non-primary 2023-12-10T19:14:51.136 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:14:51.137 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:14:51.137 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:05:49 2023 2023-12-10T19:14:51.137 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:05:49 2023 2023-12-10T19:14:51.137 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:05:49 2023 2023-12-10T19:14:51.137 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: protected: False 2023-12-10T19:14:51.137 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:14:51.137 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:14:51.137 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: 9885ccb8-b168-4992-a2b7-1fe906e83484 2023-12-10T19:14:51.137 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: false 2023-12-10T19:14:51.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:14:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror image_19 snap 2023-12-10T19:14:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_19 2023-12-10T19:14:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:14:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:14:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_19.export 2023-12-10T19:14:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_19.export 2023-12-10T19:14:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_19.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_19.export 2023-12-10T19:14:51.142 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export image_19@snap /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_19.export 2023-12-10T19:14:51.335 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:14:51.343 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export image_19@snap /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_19.export 2023-12-10T19:14:51.678 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:14:51.694 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_19.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_19.export 2023-12-10T19:14:51.773 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_19.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_19.export 2023-12-10T19:14:51.813 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:14:51.813 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_20 2023-12-10T19:14:51.813 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror image_20 snap 2023-12-10T19:14:51.813 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:14:51.813 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:51.813 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_20 2023-12-10T19:14:51.813 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:14:51.813 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create image_20@snap --debug-rbd=20 --debug-journaler=20 2023-12-10T19:14:52.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_started cluster1 mirror image_20 2023-12-10T19:14:52.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:52.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:52.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_20 2023-12-10T19:14:52.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_state cluster1 mirror image_20 started 2023-12-10T19:14:52.687 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:52.687 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:52.687 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_20 2023-12-10T19:14:52.687 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=started 2023-12-10T19:14:52.687 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:14:52.687 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:14:53.687 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_replay_state cluster1 mirror image_20 started 2023-12-10T19:14:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_20 2023-12-10T19:14:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=started 2023-12-10T19:14:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_result 2023-12-10T19:14:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=stopped 2023-12-10T19:14:53.689 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_20 2023-12-10T19:14:53.689 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_instance=cluster1 2023-12-10T19:14:53.689 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ + shift 2023-12-10T19:14:53.689 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:53.689 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:grep -i state 2023-12-10T19:14:53.689 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance=cluster1 2023-12-10T19:14:53.689 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loop_instance 2023-12-10T19:14:53.689 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:14:53.690 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T19:14:53.690 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T19:14:53.690 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_20 2023-12-10T19:14:53.691 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T19:14:53.691 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:14:53.691 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:14:53.691 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T19:14:53.691 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:14:53.691 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:14:53.691 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:14:53.691 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:14:53.691 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:14:53.691 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:14:53.691 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:14:53.691 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:14:53.691 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:14:53.691 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:14:53.692 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T19:14:53.693 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T19:14:53.693 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:53.693 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:14:53.693 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T19:14:53.693 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:14:53.693 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:14:53.693 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:14:53.693 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:14:53.693 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:14:53.693 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:14:53.693 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:14:53.693 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:14:53.693 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:14:53.693 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:14:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T19:14:53.718 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:14:53.718 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:14:53.718 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:14:53.718 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_20 2023-12-10T19:14:53.846 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:14:53.847 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ status_result= "state": "Replaying" 2023-12-10T19:14:53.847 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo "state": "Replaying" 2023-12-10T19:14:53.847 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep -i Replaying 2023-12-10T19:14:53.848 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "state": "Replaying" 2023-12-10T19:14:53.849 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=started 2023-12-10T19:14:53.849 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test started = started 2023-12-10T19:14:53.849 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:14:53.849 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_20 2023-12-10T19:14:53.849 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:14:53.849 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:14:53.849 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:53.849 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_20 2023-12-10T19:14:53.850 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = journal ] 2023-12-10T19:14:53.850 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = snapshot ] 2023-12-10T19:14:53.850 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_20 2023-12-10T19:14:53.850 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:14:53.850 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:14:53.850 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:53.850 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_20 2023-12-10T19:14:53.850 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster2-mirror-image_20.status 2023-12-10T19:14:53.851 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2-mirror-image_20.status 2023-12-10T19:14:53.851 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:14:53.851 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_20.status 2023-12-10T19:14:53.851 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-image_20.status 2023-12-10T19:14:53.851 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-image_20.status 2023-12-10T19:14:53.851 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:14:53.852 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_20.status 2023-12-10T19:14:53.852 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mirror_image_snapshot cluster2 mirror image_20 2023-12-10T19:14:53.852 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:14:53.852 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:53.852 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_20 2023-12-10T19:14:53.852 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_20 2023-12-10T19:14:54.722 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Snapshot ID: 157 2023-12-10T19:14:54.735 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_20 /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_20.status 2023-12-10T19:14:54.736 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:14:54.736 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:54.736 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_20 2023-12-10T19:14:54.736 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_20.status 2023-12-10T19:14:54.736 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_20 --format xml 2023-12-10T19:14:54.736 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:14:54.793 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-10T19:14:54.795 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snapshot_id=157 2023-12-10T19:14:54.795 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:14:54.796 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.2 2023-12-10T19:14:54.997 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_20.status 2023-12-10T19:14:54.997 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:54.997 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:54.997 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_20 2023-12-10T19:14:54.997 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_20.status 2023-12-10T19:14:54.997 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2023-12-10T19:14:54.997 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:14:55.064 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:14:55.067 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=87 2023-12-10T19:14:55.067 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 157 = 87 2023-12-10T19:14:55.067 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.4 2023-12-10T19:14:55.469 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_20.status 2023-12-10T19:14:55.469 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:55.469 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:55.469 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_20 2023-12-10T19:14:55.469 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_20.status 2023-12-10T19:14:55.469 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2023-12-10T19:14:55.469 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:14:55.537 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:14:55.540 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=87 2023-12-10T19:14:55.540 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 157 = 87 2023-12-10T19:14:55.540 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.8 2023-12-10T19:14:56.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_20.status 2023-12-10T19:14:56.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:56.342 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:56.342 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_20 2023-12-10T19:14:56.342 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_20.status 2023-12-10T19:14:56.342 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2023-12-10T19:14:56.342 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:14:56.411 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:14:56.412 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:14:56.414 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:14:56.414 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:14:56.414 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:14:56.414 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:14:56.414 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 157 = 2023-12-10T19:14:56.414 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1.6 2023-12-10T19:14:58.015 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_20.status 2023-12-10T19:14:58.015 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:58.015 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:58.015 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_20 2023-12-10T19:14:58.015 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_20.status 2023-12-10T19:14:58.016 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2023-12-10T19:14:58.016 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:14:58.084 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:14:58.088 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=157 2023-12-10T19:14:58.088 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 157 = 157 2023-12-10T19:14:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:14:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snap_present cluster1 mirror image_20 snap 2023-12-10T19:14:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:14:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_20 2023-12-10T19:14:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:14:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:14:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:14:59.089 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 info mirror/image_20@snap 2023-12-10T19:14:59.151 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_20': 2023-12-10T19:14:59.151 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:14:59.151 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:14:59.151 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 2 2023-12-10T19:14:59.151 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 128b718e08dc 2023-12-10T19:14:59.151 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.128b718e08dc 2023-12-10T19:14:59.151 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:14:59.152 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering, non-primary 2023-12-10T19:14:59.152 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:14:59.152 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:14:59.152 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:06:04 2023 2023-12-10T19:14:59.152 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:06:04 2023 2023-12-10T19:14:59.152 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:06:04 2023 2023-12-10T19:14:59.152 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: protected: False 2023-12-10T19:14:59.152 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:14:59.152 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:14:59.152 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: ad3d186e-bfbd-40cc-b37d-73be0b0f0a7d 2023-12-10T19:14:59.152 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: false 2023-12-10T19:14:59.155 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:14:59.155 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror image_20 snap 2023-12-10T19:14:59.155 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:59.155 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_20 2023-12-10T19:14:59.156 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:14:59.156 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:14:59.156 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_20.export 2023-12-10T19:14:59.156 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_20.export 2023-12-10T19:14:59.156 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_20.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_20.export 2023-12-10T19:14:59.156 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export image_20@snap /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_20.export 2023-12-10T19:14:59.379 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:14:59.391 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export image_20@snap /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_20.export 2023-12-10T19:14:59.765 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:14:59.781 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_20.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_20.export 2023-12-10T19:14:59.857 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_20.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_20.export 2023-12-10T19:14:59.896 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:14:59.897 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_21 2023-12-10T19:14:59.897 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror image_21 snap 2023-12-10T19:14:59.897 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:14:59.897 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:14:59.897 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_21 2023-12-10T19:14:59.897 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:14:59.897 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create image_21@snap --debug-rbd=20 --debug-journaler=20 2023-12-10T19:15:00.720 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_started cluster1 mirror image_21 2023-12-10T19:15:00.720 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:00.721 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:00.721 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_21 2023-12-10T19:15:00.721 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_state cluster1 mirror image_21 started 2023-12-10T19:15:00.721 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:00.721 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:00.721 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_21 2023-12-10T19:15:00.721 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=started 2023-12-10T19:15:00.721 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:15:00.721 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:15:01.722 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_replay_state cluster1 mirror image_21 started 2023-12-10T19:15:01.722 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:01.722 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:01.722 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_21 2023-12-10T19:15:01.722 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=started 2023-12-10T19:15:01.722 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_result 2023-12-10T19:15:01.722 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=stopped 2023-12-10T19:15:01.723 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_21 2023-12-10T19:15:01.723 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ + local cluster_instance=cluster1 2023-12-10T19:15:01.723 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:15:01.723 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:grep -i state 2023-12-10T19:15:01.723 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:01.723 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance=cluster1 2023-12-10T19:15:01.724 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loop_instance 2023-12-10T19:15:01.724 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:15:01.725 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T19:15:01.726 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T19:15:01.726 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_21 2023-12-10T19:15:01.726 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T19:15:01.726 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:15:01.726 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:15:01.726 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T19:15:01.726 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:15:01.726 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:15:01.726 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:15:01.726 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:01.726 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:15:01.726 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:15:01.726 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:15:01.726 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:01.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:15:01.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:15:01.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T19:15:01.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T19:15:01.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:01.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:15:01.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T19:15:01.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:15:01.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:15:01.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:15:01.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:01.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:15:01.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:15:01.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:15:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:15:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:15:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:15:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T19:15:01.747 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:15:01.747 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:15:01.748 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:15:01.748 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_21 2023-12-10T19:15:01.875 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:15:01.876 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ status_result= "state": "Replaying" 2023-12-10T19:15:01.876 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo "state": "Replaying" 2023-12-10T19:15:01.876 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep -i Replaying 2023-12-10T19:15:01.877 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "state": "Replaying" 2023-12-10T19:15:01.878 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=started 2023-12-10T19:15:01.878 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test started = started 2023-12-10T19:15:01.878 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:15:01.878 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_21 2023-12-10T19:15:01.879 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:15:01.879 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:15:01.879 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:01.879 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_21 2023-12-10T19:15:01.879 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = journal ] 2023-12-10T19:15:01.879 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = snapshot ] 2023-12-10T19:15:01.879 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_21 2023-12-10T19:15:01.879 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:15:01.879 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:15:01.879 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:01.879 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_21 2023-12-10T19:15:01.879 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster2-mirror-image_21.status 2023-12-10T19:15:01.879 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2-mirror-image_21.status 2023-12-10T19:15:01.879 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:15:01.880 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_21.status 2023-12-10T19:15:01.881 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-image_21.status 2023-12-10T19:15:01.881 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-image_21.status 2023-12-10T19:15:01.881 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:15:01.881 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_21.status 2023-12-10T19:15:01.881 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mirror_image_snapshot cluster2 mirror image_21 2023-12-10T19:15:01.881 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:15:01.881 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:01.881 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_21 2023-12-10T19:15:01.881 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_21 2023-12-10T19:15:02.749 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Snapshot ID: 159 2023-12-10T19:15:02.760 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_21 /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_21.status 2023-12-10T19:15:02.760 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:15:02.760 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:02.761 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_21 2023-12-10T19:15:02.761 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_21.status 2023-12-10T19:15:02.761 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_21 --format xml 2023-12-10T19:15:02.761 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:15:02.818 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-10T19:15:02.820 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snapshot_id=159 2023-12-10T19:15:02.820 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:15:02.820 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.2 2023-12-10T19:15:03.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_21.status 2023-12-10T19:15:03.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:03.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:03.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_21 2023-12-10T19:15:03.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_21.status 2023-12-10T19:15:03.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2023-12-10T19:15:03.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:15:03.095 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:15:03.097 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=88 2023-12-10T19:15:03.097 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 159 = 88 2023-12-10T19:15:03.097 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.4 2023-12-10T19:15:03.498 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_21.status 2023-12-10T19:15:03.498 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:03.498 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:03.498 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_21 2023-12-10T19:15:03.498 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_21.status 2023-12-10T19:15:03.499 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2023-12-10T19:15:03.499 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:15:03.567 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:15:03.570 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=88 2023-12-10T19:15:03.570 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 159 = 88 2023-12-10T19:15:03.570 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.8 2023-12-10T19:15:04.371 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_21.status 2023-12-10T19:15:04.371 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:04.371 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:04.371 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_21 2023-12-10T19:15:04.371 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_21.status 2023-12-10T19:15:04.371 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2023-12-10T19:15:04.372 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:15:04.440 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:15:04.440 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:15:04.442 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:15:04.442 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:15:04.442 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:15:04.443 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:15:04.443 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 159 = 2023-12-10T19:15:04.443 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1.6 2023-12-10T19:15:06.044 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_21.status 2023-12-10T19:15:06.044 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:06.044 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:06.044 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_21 2023-12-10T19:15:06.044 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_21.status 2023-12-10T19:15:06.044 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2023-12-10T19:15:06.045 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:15:06.114 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:15:06.117 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=159 2023-12-10T19:15:06.117 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 159 = 159 2023-12-10T19:15:06.117 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:15:06.117 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snap_present cluster1 mirror image_21 snap 2023-12-10T19:15:06.117 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:06.117 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:06.117 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_21 2023-12-10T19:15:06.117 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:15:06.117 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:15:06.117 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:15:07.118 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 info mirror/image_21@snap 2023-12-10T19:15:07.180 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_21': 2023-12-10T19:15:07.180 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:15:07.180 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:15:07.180 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 2 2023-12-10T19:15:07.180 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 10c08106a248 2023-12-10T19:15:07.181 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.10c08106a248 2023-12-10T19:15:07.181 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:15:07.181 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering, non-primary 2023-12-10T19:15:07.181 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:15:07.181 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:15:07.181 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:06:15 2023 2023-12-10T19:15:07.181 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:06:15 2023 2023-12-10T19:15:07.181 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:06:15 2023 2023-12-10T19:15:07.181 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: protected: False 2023-12-10T19:15:07.181 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:15:07.181 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:15:07.181 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: 341379c3-72c1-42e0-8bda-bc8752bf5c19 2023-12-10T19:15:07.182 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: false 2023-12-10T19:15:07.184 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:15:07.185 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror image_21 snap 2023-12-10T19:15:07.185 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:07.185 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_21 2023-12-10T19:15:07.185 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:15:07.185 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:15:07.185 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_21.export 2023-12-10T19:15:07.185 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_21.export 2023-12-10T19:15:07.185 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_21.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_21.export 2023-12-10T19:15:07.185 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export image_21@snap /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_21.export 2023-12-10T19:15:07.387 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:15:07.393 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export image_21@snap /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_21.export 2023-12-10T19:15:07.662 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:15:07.679 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_21.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_21.export 2023-12-10T19:15:07.756 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_21.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_21.export 2023-12-10T19:15:07.795 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:15:07.795 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_22 2023-12-10T19:15:07.795 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror image_22 snap 2023-12-10T19:15:07.795 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:15:07.795 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:07.796 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_22 2023-12-10T19:15:07.796 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:15:07.796 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create image_22@snap --debug-rbd=20 --debug-journaler=20 2023-12-10T19:15:08.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_started cluster1 mirror image_22 2023-12-10T19:15:08.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:08.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:08.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_22 2023-12-10T19:15:08.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_state cluster1 mirror image_22 started 2023-12-10T19:15:08.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:08.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:08.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_22 2023-12-10T19:15:08.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=started 2023-12-10T19:15:08.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:15:08.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:15:09.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_replay_state cluster1 mirror image_22 started 2023-12-10T19:15:09.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:09.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:09.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_22 2023-12-10T19:15:09.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=started 2023-12-10T19:15:09.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_result 2023-12-10T19:15:09.699 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=stopped 2023-12-10T19:15:09.700 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_22 2023-12-10T19:15:09.700 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_instance=cluster1 2023-12-10T19:15:09.700 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:15:09.700 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:09.700 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ + local instance=cluster1 2023-12-10T19:15:09.700 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loop_instance 2023-12-10T19:15:09.700 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleepgrep 0 2023-12-10T19:15:09.700 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: -i state 2023-12-10T19:15:09.700 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T19:15:09.700 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T19:15:09.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_22 2023-12-10T19:15:09.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T19:15:09.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:15:09.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:15:09.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T19:15:09.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:15:09.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:15:09.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:15:09.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:09.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:15:09.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:15:09.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:15:09.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:09.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:15:09.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:15:09.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T19:15:09.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T19:15:09.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:09.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:15:09.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T19:15:09.704 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:15:09.704 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:15:09.704 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:15:09.704 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:09.704 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:15:09.704 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:15:09.704 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:15:09.704 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:15:09.705 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:15:09.705 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:15:09.705 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T19:15:09.728 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:15:09.728 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:15:09.728 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:15:09.728 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_22 2023-12-10T19:15:09.844 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2023-12-10T19:15:09.844 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:15:09.844 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:15:09.847 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_22 2023-12-10T19:15:09.847 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:1 2023-12-10T19:15:09.847 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:15:09.847 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:15:09.847 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-10T19:15:09.847 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:15:09.847 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:15:09.847 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:15:09.847 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:09.847 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:15:09.848 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:15:09.848 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:15:09.848 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:09.848 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:15:09.848 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:15:09.848 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-10T19:15:09.848 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:1 2023-12-10T19:15:09.848 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:09.848 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:15:09.848 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-10T19:15:09.848 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:15:09.848 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:15:09.848 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:15:09.848 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:09.849 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:15:09.849 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:15:09.849 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:15:09.849 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:15:09.849 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:15:09.849 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:15:09.849 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-10T19:15:09.866 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:15:09.866 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:15:09.866 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:15:09.866 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_22 2023-12-10T19:15:09.974 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2023-12-10T19:15:09.974 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:15:09.974 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:15:09.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_22 2023-12-10T19:15:09.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:2 2023-12-10T19:15:09.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:15:09.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:15:09.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-10T19:15:09.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:15:09.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:15:09.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:15:09.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:09.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:15:09.978 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:15:09.978 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:15:09.978 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:09.978 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:15:09.978 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:15:09.978 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-10T19:15:09.978 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:2 2023-12-10T19:15:09.978 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:09.978 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:15:09.978 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-10T19:15:09.978 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:15:09.978 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:15:09.978 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:15:09.978 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:09.978 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:15:09.979 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:15:09.979 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:15:09.979 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:15:09.979 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:15:09.979 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:15:09.979 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-10T19:15:10.003 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:15:10.003 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:15:10.003 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:15:10.003 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_22 2023-12-10T19:15:10.128 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2023-12-10T19:15:10.128 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:15:10.128 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:15:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_22 2023-12-10T19:15:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:3 2023-12-10T19:15:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:15:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:15:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-10T19:15:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:3 2023-12-10T19:15:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:15:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:15:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:15:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 3 = cluster1:3 ] 2023-12-10T19:15:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:15:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=3 2023-12-10T19:15:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:15:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-10T19:15:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:3 2023-12-10T19:15:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:15:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-10T19:15:10.132 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:3 2023-12-10T19:15:10.132 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:15:10.132 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:15:10.132 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:10.132 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:15:10.132 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 3 = cluster1:3 ] 2023-12-10T19:15:10.132 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:15:10.132 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:15:10.132 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=3 2023-12-10T19:15:10.132 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:15:10.132 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-10T19:15:10.154 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T19:15:10.154 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T19:15:10.154 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T19:15:10.154 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_22 2023-12-10T19:15:10.286 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:15:10.287 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ status_result= "state": "Replaying" 2023-12-10T19:15:10.287 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo "state": "Replaying" 2023-12-10T19:15:10.287 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep -i Replaying 2023-12-10T19:15:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "state": "Replaying" 2023-12-10T19:15:10.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=started 2023-12-10T19:15:10.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test started = started 2023-12-10T19:15:10.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:15:10.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_22 2023-12-10T19:15:10.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:15:10.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:15:10.290 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:10.290 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_22 2023-12-10T19:15:10.290 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = journal ] 2023-12-10T19:15:10.290 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = snapshot ] 2023-12-10T19:15:10.290 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_22 2023-12-10T19:15:10.290 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:15:10.290 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:15:10.290 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:10.290 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_22 2023-12-10T19:15:10.290 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster2-mirror-image_22.status 2023-12-10T19:15:10.290 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2-mirror-image_22.status 2023-12-10T19:15:10.290 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:15:10.291 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_22.status 2023-12-10T19:15:10.291 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-image_22.status 2023-12-10T19:15:10.291 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-image_22.status 2023-12-10T19:15:10.292 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:15:10.292 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_22.status 2023-12-10T19:15:10.292 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mirror_image_snapshot cluster2 mirror image_22 2023-12-10T19:15:10.292 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:15:10.292 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:10.292 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_22 2023-12-10T19:15:10.293 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_22 2023-12-10T19:15:10.721 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Snapshot ID: 161 2023-12-10T19:15:10.735 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_22 /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_22.status 2023-12-10T19:15:10.735 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:15:10.735 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:10.735 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_22 2023-12-10T19:15:10.735 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_22.status 2023-12-10T19:15:10.735 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_22 --format xml 2023-12-10T19:15:10.735 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:15:11.003 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-10T19:15:11.006 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snapshot_id=161 2023-12-10T19:15:11.006 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:15:11.006 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.2 2023-12-10T19:15:11.208 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_22.status 2023-12-10T19:15:11.208 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:11.208 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:11.208 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_22 2023-12-10T19:15:11.208 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_22.status 2023-12-10T19:15:11.208 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2023-12-10T19:15:11.208 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:15:11.276 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:15:11.279 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=89 2023-12-10T19:15:11.279 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 161 = 89 2023-12-10T19:15:11.279 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.4 2023-12-10T19:15:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_22.status 2023-12-10T19:15:11.681 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:11.681 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:11.681 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_22 2023-12-10T19:15:11.681 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_22.status 2023-12-10T19:15:11.681 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2023-12-10T19:15:11.681 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:15:11.758 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:15:11.759 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:15:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:15:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:15:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:15:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:15:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 161 = 2023-12-10T19:15:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.8 2023-12-10T19:15:12.563 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_22.status 2023-12-10T19:15:12.563 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:12.563 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:12.563 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_22 2023-12-10T19:15:12.563 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_22.status 2023-12-10T19:15:12.563 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2023-12-10T19:15:12.564 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:15:12.726 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:15:12.726 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:15:12.729 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:15:12.729 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:15:12.729 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:15:12.729 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:15:12.729 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 161 = 2023-12-10T19:15:12.729 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1.6 2023-12-10T19:15:14.331 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_22.status 2023-12-10T19:15:14.332 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:14.332 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:14.332 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_22 2023-12-10T19:15:14.332 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_22.status 2023-12-10T19:15:14.332 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2023-12-10T19:15:14.332 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:15:14.401 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:15:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=161 2023-12-10T19:15:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 161 = 161 2023-12-10T19:15:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:15:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snap_present cluster1 mirror image_22 snap 2023-12-10T19:15:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_22 2023-12-10T19:15:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:15:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:15:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:15:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 info mirror/image_22@snap 2023-12-10T19:15:15.467 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_22': 2023-12-10T19:15:15.467 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:15:15.467 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:15:15.467 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 2 2023-12-10T19:15:15.467 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 10e860d31de4 2023-12-10T19:15:15.467 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.10e860d31de4 2023-12-10T19:15:15.467 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:15:15.467 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering, non-primary 2023-12-10T19:15:15.467 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:15:15.467 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:15:15.468 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:06:26 2023 2023-12-10T19:15:15.468 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:06:26 2023 2023-12-10T19:15:15.468 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:06:26 2023 2023-12-10T19:15:15.468 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: protected: False 2023-12-10T19:15:15.468 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:15:15.468 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:15:15.468 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: 5f3f027e-9f9a-463e-be32-1e7c54f42e4e 2023-12-10T19:15:15.468 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: false 2023-12-10T19:15:15.472 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:15:15.472 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror image_22 snap 2023-12-10T19:15:15.472 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:15.472 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_22 2023-12-10T19:15:15.472 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:15:15.472 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:15:15.472 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_22.export 2023-12-10T19:15:15.472 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_22.export 2023-12-10T19:15:15.472 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_22.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_22.export 2023-12-10T19:15:15.472 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export image_22@snap /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_22.export 2023-12-10T19:15:15.670 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:15:15.680 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export image_22@snap /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_22.export 2023-12-10T19:15:16.026 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:15:16.039 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_22.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_22.export 2023-12-10T19:15:16.115 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_22.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_22.export 2023-12-10T19:15:16.156 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:15:16.156 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_23 2023-12-10T19:15:16.156 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror image_23 snap 2023-12-10T19:15:16.156 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:15:16.156 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:16.157 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_23 2023-12-10T19:15:16.157 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:15:16.157 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create image_23@snap --debug-rbd=20 --debug-journaler=20 2023-12-10T19:15:16.368 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_started cluster1 mirror image_23 2023-12-10T19:15:16.368 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:16.368 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:16.368 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_23 2023-12-10T19:15:16.369 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_state cluster1 mirror image_23 started 2023-12-10T19:15:16.369 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:16.369 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:16.369 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_23 2023-12-10T19:15:16.369 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=started 2023-12-10T19:15:16.369 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:15:16.369 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:15:17.370 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_replay_state cluster1 mirror image_23 started 2023-12-10T19:15:17.370 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:17.370 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:17.370 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_23 2023-12-10T19:15:17.370 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=started 2023-12-10T19:15:17.370 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_result 2023-12-10T19:15:17.370 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=stopped 2023-12-10T19:15:17.371 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_23 2023-12-10T19:15:17.372 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ + local cluster_instance=cluster1 2023-12-10T19:15:17.372 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:15:17.372 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:grep -i state 2023-12-10T19:15:17.372 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:17.372 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance=cluster1 2023-12-10T19:15:17.372 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loop_instance 2023-12-10T19:15:17.372 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:15:17.372 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T19:15:17.372 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T19:15:17.374 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_23 2023-12-10T19:15:17.374 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T19:15:17.374 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:15:17.374 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:15:17.374 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T19:15:17.374 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:15:17.374 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:15:17.374 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:15:17.374 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:17.374 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:15:17.374 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:15:17.374 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:15:17.375 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:17.375 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:15:17.375 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:15:17.375 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T19:15:17.375 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T19:15:17.375 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:17.375 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:15:17.375 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T19:15:17.375 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:15:17.375 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:15:17.375 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:15:17.375 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:17.375 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:15:17.376 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:15:17.376 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:15:17.376 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:15:17.376 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:15:17.376 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:15:17.376 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T19:15:17.398 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:15:17.398 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:15:17.399 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:15:17.399 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_23 2023-12-10T19:15:17.510 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2023-12-10T19:15:17.510 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:15:17.510 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:15:17.512 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_23 2023-12-10T19:15:17.512 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:1 2023-12-10T19:15:17.512 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:15:17.512 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:15:17.512 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-10T19:15:17.512 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:15:17.512 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:15:17.512 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:15:17.512 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:17.513 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:15:17.513 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:15:17.513 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:15:17.513 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:17.513 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:15:17.513 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:15:17.513 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-10T19:15:17.513 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:1 2023-12-10T19:15:17.513 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:17.513 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:15:17.513 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-10T19:15:17.513 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:15:17.514 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:15:17.515 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:15:17.515 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:17.515 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:15:17.515 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:15:17.515 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:15:17.515 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:15:17.515 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:15:17.515 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:15:17.515 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-10T19:15:17.537 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:15:17.537 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:15:17.537 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:15:17.537 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_23 2023-12-10T19:15:17.657 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2023-12-10T19:15:17.658 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:15:17.658 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:15:17.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_23 2023-12-10T19:15:17.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:2 2023-12-10T19:15:17.661 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:15:17.661 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:15:17.661 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-10T19:15:17.661 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:15:17.661 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:15:17.661 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:15:17.661 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:17.661 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:15:17.661 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:15:17.661 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:15:17.661 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:17.661 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:15:17.661 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:15:17.661 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-10T19:15:17.661 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:2 2023-12-10T19:15:17.661 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:17.662 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:15:17.662 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-10T19:15:17.662 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:15:17.662 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:15:17.662 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:15:17.662 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:17.662 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:15:17.662 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:15:17.662 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:15:17.662 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:15:17.662 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:15:17.662 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:15:17.662 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-10T19:15:17.687 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:15:17.687 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:15:17.687 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:15:17.687 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_23 2023-12-10T19:15:17.815 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:15:17.816 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ status_result= "state": "Replaying" 2023-12-10T19:15:17.816 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo "state": "Replaying" 2023-12-10T19:15:17.816 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep -i Replaying 2023-12-10T19:15:17.817 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "state": "Replaying" 2023-12-10T19:15:17.820 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=started 2023-12-10T19:15:17.820 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test started = started 2023-12-10T19:15:17.820 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:15:17.820 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_23 2023-12-10T19:15:17.821 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:15:17.821 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:15:17.821 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:17.821 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_23 2023-12-10T19:15:17.821 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = journal ] 2023-12-10T19:15:17.821 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = snapshot ] 2023-12-10T19:15:17.821 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_23 2023-12-10T19:15:17.821 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:15:17.821 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:15:17.821 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:17.821 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_23 2023-12-10T19:15:17.821 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster2-mirror-image_23.status 2023-12-10T19:15:17.821 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2-mirror-image_23.status 2023-12-10T19:15:17.821 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:15:17.821 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_23.status 2023-12-10T19:15:17.822 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-image_23.status 2023-12-10T19:15:17.822 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-image_23.status 2023-12-10T19:15:17.822 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:15:17.822 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_23.status 2023-12-10T19:15:17.822 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mirror_image_snapshot cluster2 mirror image_23 2023-12-10T19:15:17.822 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:15:17.822 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:17.823 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_23 2023-12-10T19:15:17.823 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_23 2023-12-10T19:15:18.405 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Snapshot ID: 163 2023-12-10T19:15:18.418 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_23 /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_23.status 2023-12-10T19:15:18.418 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:15:18.418 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:18.418 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_23 2023-12-10T19:15:18.419 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_23.status 2023-12-10T19:15:18.419 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_23 --format xml 2023-12-10T19:15:18.419 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:15:18.683 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-10T19:15:18.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snapshot_id=163 2023-12-10T19:15:18.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:15:18.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.2 2023-12-10T19:15:18.888 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_23.status 2023-12-10T19:15:18.888 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:18.888 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:18.888 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_23 2023-12-10T19:15:18.888 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_23.status 2023-12-10T19:15:18.888 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2023-12-10T19:15:18.888 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:15:18.962 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:15:18.962 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:15:18.964 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:15:18.964 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:15:18.964 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:15:18.965 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:15:18.965 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 163 = 2023-12-10T19:15:18.965 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.4 2023-12-10T19:15:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_23.status 2023-12-10T19:15:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:19.367 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_23 2023-12-10T19:15:19.367 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_23.status 2023-12-10T19:15:19.367 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2023-12-10T19:15:19.367 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:15:19.438 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:15:19.439 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:15:19.441 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:15:19.441 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:15:19.441 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:15:19.441 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:15:19.441 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 163 = 2023-12-10T19:15:19.441 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.8 2023-12-10T19:15:20.246 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_23.status 2023-12-10T19:15:20.246 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:20.246 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:20.246 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_23 2023-12-10T19:15:20.246 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_23.status 2023-12-10T19:15:20.247 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2023-12-10T19:15:20.247 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:15:20.313 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:15:20.315 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=163 2023-12-10T19:15:20.315 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 163 = 163 2023-12-10T19:15:20.315 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:15:20.315 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snap_present cluster1 mirror image_23 snap 2023-12-10T19:15:20.315 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:20.315 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_23 2023-12-10T19:15:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:15:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:15:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:15:21.316 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 info mirror/image_23@snap 2023-12-10T19:15:21.374 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_23': 2023-12-10T19:15:21.374 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:15:21.374 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:15:21.374 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 2 2023-12-10T19:15:21.375 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 129bd33880ee 2023-12-10T19:15:21.375 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.129bd33880ee 2023-12-10T19:15:21.375 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:15:21.375 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering, non-primary 2023-12-10T19:15:21.375 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:15:21.375 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:15:21.375 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:06:38 2023 2023-12-10T19:15:21.375 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:06:38 2023 2023-12-10T19:15:21.376 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:06:38 2023 2023-12-10T19:15:21.376 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: protected: False 2023-12-10T19:15:21.376 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:15:21.376 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:15:21.376 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: b1ff34f6-63f9-49d9-9315-4eabc97c393e 2023-12-10T19:15:21.376 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: false 2023-12-10T19:15:21.378 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:15:21.379 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror image_23 snap 2023-12-10T19:15:21.379 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:21.379 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_23 2023-12-10T19:15:21.379 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:15:21.379 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:15:21.379 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_23.export 2023-12-10T19:15:21.379 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_23.export 2023-12-10T19:15:21.379 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_23.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_23.export 2023-12-10T19:15:21.379 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export image_23@snap /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_23.export 2023-12-10T19:15:21.581 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:15:21.589 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export image_23@snap /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_23.export 2023-12-10T19:15:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:15:21.834 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_23.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_23.export 2023-12-10T19:15:21.909 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_23.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_23.export 2023-12-10T19:15:21.950 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:15:21.950 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_24 2023-12-10T19:15:21.950 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror image_24 snap 2023-12-10T19:15:21.950 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:15:21.950 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:21.951 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_24 2023-12-10T19:15:21.951 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:15:21.951 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create image_24@snap --debug-rbd=20 --debug-journaler=20 2023-12-10T19:15:22.700 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_started cluster1 mirror image_24 2023-12-10T19:15:22.700 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:22.700 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:22.700 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_24 2023-12-10T19:15:22.701 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_state cluster1 mirror image_24 started 2023-12-10T19:15:22.701 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:22.701 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:22.701 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_24 2023-12-10T19:15:22.701 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=started 2023-12-10T19:15:22.701 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:15:22.701 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:15:23.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_replay_state cluster1 mirror image_24 started 2023-12-10T19:15:23.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:23.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:23.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_24 2023-12-10T19:15:23.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=started 2023-12-10T19:15:23.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_result 2023-12-10T19:15:23.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=stopped 2023-12-10T19:15:23.704 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_24 2023-12-10T19:15:23.704 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_instance=cluster1 2023-12-10T19:15:23.704 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:15:23.704 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:23.704 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance=cluster1 2023-12-10T19:15:23.705 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loop_instance 2023-12-10T19:15:23.705 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:15:23.705 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep -i state 2023-12-10T19:15:23.705 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T19:15:23.705 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T19:15:23.707 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_24 2023-12-10T19:15:23.707 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T19:15:23.707 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:15:23.707 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:15:23.708 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T19:15:23.708 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:15:23.708 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:15:23.708 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:15:23.708 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:23.708 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:15:23.709 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:15:23.709 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:15:23.709 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:23.709 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:15:23.709 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:15:23.709 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T19:15:23.709 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T19:15:23.710 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:23.710 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:15:23.710 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T19:15:23.710 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:15:23.710 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:15:23.710 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:15:23.711 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:23.711 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:15:23.711 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:15:23.711 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:15:23.711 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:15:23.711 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:15:23.711 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:15:23.711 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T19:15:23.728 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:15:23.728 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:15:23.728 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:15:23.728 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_24 2023-12-10T19:15:23.859 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:15:23.860 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ status_result= "state": "Replaying" 2023-12-10T19:15:23.860 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo "state": "Replaying" 2023-12-10T19:15:23.860 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep -i Replaying 2023-12-10T19:15:23.860 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "state": "Replaying" 2023-12-10T19:15:23.862 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=started 2023-12-10T19:15:23.862 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test started = started 2023-12-10T19:15:23.862 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:15:23.862 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_24 2023-12-10T19:15:23.862 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:15:23.862 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:15:23.862 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:23.862 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_24 2023-12-10T19:15:23.862 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = journal ] 2023-12-10T19:15:23.862 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = snapshot ] 2023-12-10T19:15:23.862 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_24 2023-12-10T19:15:23.864 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:15:23.864 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:15:23.864 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:23.864 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_24 2023-12-10T19:15:23.864 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster2-mirror-image_24.status 2023-12-10T19:15:23.864 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2-mirror-image_24.status 2023-12-10T19:15:23.864 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:15:23.864 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_24.status 2023-12-10T19:15:23.864 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-image_24.status 2023-12-10T19:15:23.865 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-image_24.status 2023-12-10T19:15:23.865 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:15:23.866 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_24.status 2023-12-10T19:15:23.866 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mirror_image_snapshot cluster2 mirror image_24 2023-12-10T19:15:23.866 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:15:23.866 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:23.866 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_24 2023-12-10T19:15:23.867 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_24 2023-12-10T19:15:24.720 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Snapshot ID: 165 2023-12-10T19:15:24.735 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_24 /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_24.status 2023-12-10T19:15:24.735 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:15:24.735 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:24.735 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_24 2023-12-10T19:15:24.735 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_24.status 2023-12-10T19:15:24.735 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_24 --format xml 2023-12-10T19:15:24.735 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:15:24.796 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-10T19:15:24.798 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snapshot_id=165 2023-12-10T19:15:24.798 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:15:24.798 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.2 2023-12-10T19:15:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_24.status 2023-12-10T19:15:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:25.000 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:25.000 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_24 2023-12-10T19:15:25.000 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_24.status 2023-12-10T19:15:25.000 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2023-12-10T19:15:25.000 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:15:25.071 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:15:25.071 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:15:25.074 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:15:25.074 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:15:25.074 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:15:25.074 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:15:25.074 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 165 = 2023-12-10T19:15:25.074 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.4 2023-12-10T19:15:25.476 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_24.status 2023-12-10T19:15:25.476 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:25.476 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:25.476 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_24 2023-12-10T19:15:25.476 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_24.status 2023-12-10T19:15:25.476 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2023-12-10T19:15:25.476 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:15:25.547 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:15:25.547 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:15:25.549 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:15:25.549 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:15:25.549 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:15:25.549 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:15:25.550 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 165 = 2023-12-10T19:15:25.550 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.8 2023-12-10T19:15:25.868 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.4981661892157353) > weight (0.25) 2023-12-10T19:15:25.868 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.3660404846926921) > weight (0.25) 2023-12-10T19:15:25.868 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2023-12-10T19:15:25.868 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2023-12-10T19:15:25.869 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2023-12-10T19:15:25.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi076.stderr:2023-12-10T19:15:25.873+0000 7f6182aed640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 41064) UID: 1000 2023-12-10T19:15:26.351 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_24.status 2023-12-10T19:15:26.351 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:26.351 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:26.351 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_24 2023-12-10T19:15:26.351 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_24.status 2023-12-10T19:15:26.351 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2023-12-10T19:15:26.351 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:15:26.422 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:15:26.425 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=165 2023-12-10T19:15:26.425 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 165 = 165 2023-12-10T19:15:26.425 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:15:26.425 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snap_present cluster1 mirror image_24 snap 2023-12-10T19:15:26.425 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:26.425 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:26.425 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_24 2023-12-10T19:15:26.425 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:15:26.425 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:15:26.425 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:15:27.340 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr:2023-12-10T19:15:27.345+0000 7f924e25c640 -1 rbd::mirror::ImageReplayer: 0x559dd7f9f900 [3/2e46723e-74b1-4e29-b6e7-8617064dd725] start: already running 2023-12-10T19:15:27.340 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr:2023-12-10T19:15:27.345+0000 7f924e25c640 -1 rbd::mirror::ImageReplayer: 0x559dd7f9f900 [3/2e46723e-74b1-4e29-b6e7-8617064dd725] start: already running 2023-12-10T19:15:27.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr:2023-12-10T19:15:27.353+0000 7f924da5b640 -1 rbd::mirror::ImageReplayer: 0x559dd7f9e500 [3/ad3d186e-bfbd-40cc-b37d-73be0b0f0a7d] start: already running 2023-12-10T19:15:27.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr:2023-12-10T19:15:27.353+0000 7f924da5b640 -1 rbd::mirror::ImageReplayer: 0x559dd7f9fb80 [3/8fd23019-e0bb-4243-ad49-4fbc841c8b88] start: already running 2023-12-10T19:15:27.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr:2023-12-10T19:15:27.353+0000 7f924da5b640 -1 rbd::mirror::ImageReplayer: 0x559dd7f9ef00 [3/13cd2888-c5a1-4f79-8bc5-a118e5cace49] start: already running 2023-12-10T19:15:27.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr:2023-12-10T19:15:27.353+0000 7f924da5b640 -1 rbd::mirror::ImageReplayer: 0x559dd7f9ef00 [3/13cd2888-c5a1-4f79-8bc5-a118e5cace49] start: already running 2023-12-10T19:15:27.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr:2023-12-10T19:15:27.353+0000 7f924da5b640 -1 rbd::mirror::ImageReplayer: 0x559dd7f9f180 [3/2920a7c1-c4ea-4349-87ad-88428404bcc1] start: already running 2023-12-10T19:15:27.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr:2023-12-10T19:15:27.353+0000 7f924da5b640 -1 rbd::mirror::ImageReplayer: 0x559dd7f9f180 [3/2920a7c1-c4ea-4349-87ad-88428404bcc1] start: already running 2023-12-10T19:15:27.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr:2023-12-10T19:15:27.353+0000 7f924da5b640 -1 rbd::mirror::ImageReplayer: 0x559dd7f9f680 [3/9885ccb8-b168-4992-a2b7-1fe906e83484] start: already running 2023-12-10T19:15:27.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr:2023-12-10T19:15:27.353+0000 7f924da5b640 -1 rbd::mirror::ImageReplayer: 0x559dd7f9e280 [3/341379c3-72c1-42e0-8bda-bc8752bf5c19] start: already running 2023-12-10T19:15:27.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr:2023-12-10T19:15:27.353+0000 7f924da5b640 -1 rbd::mirror::ImageReplayer: 0x559dd7f9e280 [3/341379c3-72c1-42e0-8bda-bc8752bf5c19] start: already running 2023-12-10T19:15:27.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr:2023-12-10T19:15:27.353+0000 7f924da5b640 -1 rbd::mirror::ImageReplayer: 0x559dd7f9f400 [3/36d7ae4a-9a61-4f12-8a89-e0b759776b33] start: already running 2023-12-10T19:15:27.425 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 info mirror/image_24@snap 2023-12-10T19:15:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_24': 2023-12-10T19:15:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:15:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:15:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 2 2023-12-10T19:15:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 129b375966ab 2023-12-10T19:15:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.129b375966ab 2023-12-10T19:15:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:15:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering, non-primary 2023-12-10T19:15:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:15:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:15:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:06:48 2023 2023-12-10T19:15:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:06:48 2023 2023-12-10T19:15:27.576 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:06:48 2023 2023-12-10T19:15:27.576 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: protected: False 2023-12-10T19:15:27.576 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:15:27.576 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:15:27.576 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: 2920a7c1-c4ea-4349-87ad-88428404bcc1 2023-12-10T19:15:27.576 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: false 2023-12-10T19:15:27.580 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:15:27.580 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror image_24 snap 2023-12-10T19:15:27.580 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:27.580 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_24 2023-12-10T19:15:27.580 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:15:27.580 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:15:27.580 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_24.export 2023-12-10T19:15:27.580 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_24.export 2023-12-10T19:15:27.580 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_24.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_24.export 2023-12-10T19:15:27.581 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export image_24@snap /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_24.export 2023-12-10T19:15:27.787 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:15:27.798 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export image_24@snap /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_24.export 2023-12-10T19:15:28.100 INFO:tasks.ceph.cluster2.osd.2.smithi076.stderr:2023-12-10T19:15:28.097+0000 7f7f6add3640 -1 reset not still connected to 0x56430f0c1450 2023-12-10T19:15:28.100 INFO:tasks.ceph.cluster2.osd.2.smithi076.stderr:2023-12-10T19:15:28.101+0000 7f7f6add3640 -1 reset not still connected to 0x56431d120270 2023-12-10T19:15:28.100 INFO:tasks.ceph.cluster2.osd.2.smithi076.stderr:2023-12-10T19:15:28.105+0000 7f7f6add3640 -1 reset not still connected to 0x56432f2afd40 2023-12-10T19:15:28.136 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:15:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_24.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_24.export 2023-12-10T19:15:28.227 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_24.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_24.export 2023-12-10T19:15:28.267 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:15:28.267 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_25 2023-12-10T19:15:28.267 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror image_25 snap 2023-12-10T19:15:28.267 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:15:28.267 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:28.267 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_25 2023-12-10T19:15:28.267 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:15:28.267 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create image_25@snap --debug-rbd=20 --debug-journaler=20 2023-12-10T19:15:28.739 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_started cluster1 mirror image_25 2023-12-10T19:15:28.739 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:28.739 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:28.739 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_25 2023-12-10T19:15:28.739 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_state cluster1 mirror image_25 started 2023-12-10T19:15:28.739 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:28.739 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:28.739 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_25 2023-12-10T19:15:28.739 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=started 2023-12-10T19:15:28.739 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:15:28.739 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:15:29.740 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_replay_state cluster1 mirror image_25 started 2023-12-10T19:15:29.740 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:29.740 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:29.740 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_25 2023-12-10T19:15:29.740 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=started 2023-12-10T19:15:29.740 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_result 2023-12-10T19:15:29.740 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=stopped 2023-12-10T19:15:29.741 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_25 2023-12-10T19:15:29.741 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_instance=cluster1 2023-12-10T19:15:29.741 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ + shift 2023-12-10T19:15:29.741 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:29.741 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:grep -i state 2023-12-10T19:15:29.741 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance=cluster1 2023-12-10T19:15:29.741 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loop_instance 2023-12-10T19:15:29.741 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:15:29.744 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T19:15:29.744 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T19:15:29.744 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_25 2023-12-10T19:15:29.744 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T19:15:29.744 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:15:29.744 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:15:29.744 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T19:15:29.744 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:15:29.744 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:15:29.744 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:15:29.744 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:29.744 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:15:29.744 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:15:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:15:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:15:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:15:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T19:15:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T19:15:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:15:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T19:15:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:15:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:15:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:15:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:15:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:15:29.746 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:15:29.746 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:15:29.746 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:15:29.746 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:15:29.746 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T19:15:29.765 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:15:29.765 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:15:29.765 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:15:29.765 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_25 2023-12-10T19:15:29.894 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:15:29.894 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ status_result= "state": "Replaying" 2023-12-10T19:15:29.894 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo "state": "Replaying" 2023-12-10T19:15:29.894 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep -i Replaying 2023-12-10T19:15:29.895 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "state": "Replaying" 2023-12-10T19:15:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=started 2023-12-10T19:15:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test started = started 2023-12-10T19:15:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:15:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_25 2023-12-10T19:15:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:15:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:15:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_25 2023-12-10T19:15:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = journal ] 2023-12-10T19:15:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = snapshot ] 2023-12-10T19:15:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_25 2023-12-10T19:15:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:15:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:15:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_25 2023-12-10T19:15:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster2-mirror-image_25.status 2023-12-10T19:15:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2-mirror-image_25.status 2023-12-10T19:15:29.898 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:15:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_25.status 2023-12-10T19:15:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-image_25.status 2023-12-10T19:15:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-image_25.status 2023-12-10T19:15:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:15:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_25.status 2023-12-10T19:15:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mirror_image_snapshot cluster2 mirror image_25 2023-12-10T19:15:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:15:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_25 2023-12-10T19:15:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_25 2023-12-10T19:15:30.747 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Snapshot ID: 167 2023-12-10T19:15:30.761 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_25 /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_25.status 2023-12-10T19:15:30.761 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:15:30.761 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:30.761 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_25 2023-12-10T19:15:30.761 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_25.status 2023-12-10T19:15:30.761 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_25 --format xml 2023-12-10T19:15:30.761 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:15:30.818 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-10T19:15:30.820 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snapshot_id=167 2023-12-10T19:15:30.821 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:15:30.821 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.2 2023-12-10T19:15:30.869 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2023-12-10T19:15:30.869 INFO:teuthology.orchestra.run:waiting for 600 2023-12-10T19:15:30.869 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2023-12-10T19:15:30.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2023-12-10T19:15:30.870 DEBUG:teuthology.orchestra.run.smithi076:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2023-12-10T19:15:30.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2023-12-10T19:15:30.872 INFO:tasks.rbd_mirror_thrash:waiting for 63 secs before thrashing 2023-12-10T19:15:31.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_25.status 2023-12-10T19:15:31.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:31.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:31.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_25 2023-12-10T19:15:31.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_25.status 2023-12-10T19:15:31.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2023-12-10T19:15:31.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:15:31.085 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:15:31.085 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:15:31.086 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:15:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:15:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:15:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:15:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 167 = 2023-12-10T19:15:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.4 2023-12-10T19:15:31.488 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_25.status 2023-12-10T19:15:31.488 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:31.488 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:31.488 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_25 2023-12-10T19:15:31.488 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_25.status 2023-12-10T19:15:31.489 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2023-12-10T19:15:31.489 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:15:31.558 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:15:31.558 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:15:31.560 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:15:31.560 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:15:31.561 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:15:31.561 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:15:31.561 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 167 = 2023-12-10T19:15:31.561 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.8 2023-12-10T19:15:32.362 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_25.status 2023-12-10T19:15:32.362 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:32.362 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:32.362 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_25 2023-12-10T19:15:32.362 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_25.status 2023-12-10T19:15:32.362 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2023-12-10T19:15:32.363 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:15:32.429 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:15:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=167 2023-12-10T19:15:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 167 = 167 2023-12-10T19:15:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:15:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snap_present cluster1 mirror image_25 snap 2023-12-10T19:15:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_25 2023-12-10T19:15:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:15:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:15:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:15:33.433 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 info mirror/image_25@snap 2023-12-10T19:15:33.493 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_25': 2023-12-10T19:15:33.493 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:15:33.494 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:15:33.494 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 2 2023-12-10T19:15:33.494 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 10c0514c8bc 2023-12-10T19:15:33.494 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.10c0514c8bc 2023-12-10T19:15:33.494 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:15:33.494 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering, non-primary 2023-12-10T19:15:33.494 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:15:33.494 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:15:33.494 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:07:03 2023 2023-12-10T19:15:33.494 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:07:03 2023 2023-12-10T19:15:33.494 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:07:03 2023 2023-12-10T19:15:33.494 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: protected: False 2023-12-10T19:15:33.494 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:15:33.494 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:15:33.495 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: cd328d00-062e-4168-a2e8-c67acd6bae34 2023-12-10T19:15:33.495 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: false 2023-12-10T19:15:33.498 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:15:33.498 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror image_25 snap 2023-12-10T19:15:33.498 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:33.498 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_25 2023-12-10T19:15:33.498 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:15:33.498 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:15:33.498 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_25.export 2023-12-10T19:15:33.498 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_25.export 2023-12-10T19:15:33.498 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_25.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_25.export 2023-12-10T19:15:33.498 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export image_25@snap /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_25.export 2023-12-10T19:15:33.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:15:33.712 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export image_25@snap /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_25.export 2023-12-10T19:15:34.015 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:15:34.025 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_25.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_25.export 2023-12-10T19:15:34.102 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_25.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_25.export 2023-12-10T19:15:34.141 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:15:34.141 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_26 2023-12-10T19:15:34.142 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror image_26 snap 2023-12-10T19:15:34.142 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:15:34.142 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:34.142 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_26 2023-12-10T19:15:34.142 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:15:34.142 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create image_26@snap --debug-rbd=20 --debug-journaler=20 2023-12-10T19:15:34.745 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_started cluster1 mirror image_26 2023-12-10T19:15:34.745 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:34.745 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:34.745 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_26 2023-12-10T19:15:34.745 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_state cluster1 mirror image_26 started 2023-12-10T19:15:34.745 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:34.745 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:34.745 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_26 2023-12-10T19:15:34.746 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=started 2023-12-10T19:15:34.746 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:15:34.746 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:15:35.746 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_replay_state cluster1 mirror image_26 started 2023-12-10T19:15:35.746 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:35.746 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:35.746 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_26 2023-12-10T19:15:35.746 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=started 2023-12-10T19:15:35.746 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_result 2023-12-10T19:15:35.746 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=stopped 2023-12-10T19:15:35.747 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_26 2023-12-10T19:15:35.747 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_instance=cluster1 2023-12-10T19:15:35.747 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:15:35.747 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:35.747 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ + local instance=cluster1 2023-12-10T19:15:35.747 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loop_instance 2023-12-10T19:15:35.747 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:grep -i state 2023-12-10T19:15:35.747 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:15:35.749 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T19:15:35.749 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T19:15:35.749 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_26 2023-12-10T19:15:35.749 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T19:15:35.749 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:15:35.749 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:15:35.749 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T19:15:35.749 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:15:35.749 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:15:35.749 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:15:35.749 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:35.750 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:15:35.750 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:15:35.750 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:15:35.750 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:35.750 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:15:35.750 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:15:35.750 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T19:15:35.750 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T19:15:35.750 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:35.750 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:15:35.750 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T19:15:35.750 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:15:35.750 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:15:35.750 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:15:35.750 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:35.750 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:15:35.751 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:15:35.751 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:15:35.751 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:15:35.751 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:15:35.751 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:15:35.751 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T19:15:35.776 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:15:35.776 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:15:35.776 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:15:35.776 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_26 2023-12-10T19:15:35.888 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2023-12-10T19:15:35.889 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:15:35.889 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:15:35.891 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_26 2023-12-10T19:15:35.891 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:1 2023-12-10T19:15:35.891 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:15:35.891 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:15:35.891 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-10T19:15:35.891 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:15:35.891 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:15:35.891 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:15:35.891 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:35.891 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:15:35.891 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:15:35.891 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:15:35.892 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:35.892 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:15:35.892 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:15:35.892 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-10T19:15:35.892 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:1 2023-12-10T19:15:35.892 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:35.892 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:15:35.892 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-10T19:15:35.892 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:15:35.892 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:15:35.892 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:15:35.892 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:35.892 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:15:35.893 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:15:35.893 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:15:35.893 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:15:35.893 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:15:35.893 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:15:35.893 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-10T19:15:35.914 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:15:35.915 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:15:35.915 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:15:35.915 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_26 2023-12-10T19:15:36.034 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2023-12-10T19:15:36.034 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:15:36.034 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:15:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_26 2023-12-10T19:15:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:2 2023-12-10T19:15:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:15:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:15:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-10T19:15:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:15:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:15:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:15:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:15:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:15:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:15:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:15:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:15:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-10T19:15:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:2 2023-12-10T19:15:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:15:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-10T19:15:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:15:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:15:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:15:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:15:36.039 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:15:36.039 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:15:36.039 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:15:36.039 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:15:36.039 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:15:36.039 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-10T19:15:36.061 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:15:36.061 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:15:36.061 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:15:36.061 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_26 2023-12-10T19:15:36.169 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2023-12-10T19:15:36.170 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:15:36.170 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:15:36.171 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_26 2023-12-10T19:15:36.171 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:3 2023-12-10T19:15:36.171 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:15:36.171 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:15:36.171 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-10T19:15:36.171 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:3 2023-12-10T19:15:36.171 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:15:36.171 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:15:36.171 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:36.171 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:15:36.172 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 3 = cluster1:3 ] 2023-12-10T19:15:36.172 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:15:36.172 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:36.172 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=3 2023-12-10T19:15:36.172 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:15:36.173 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-10T19:15:36.173 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:3 2023-12-10T19:15:36.173 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:36.173 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:15:36.173 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-10T19:15:36.173 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:3 2023-12-10T19:15:36.173 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:15:36.173 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:15:36.173 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:36.173 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:15:36.173 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 3 = cluster1:3 ] 2023-12-10T19:15:36.173 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:15:36.174 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:15:36.174 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=3 2023-12-10T19:15:36.174 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:15:36.174 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-10T19:15:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T19:15:36.197 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T19:15:36.197 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T19:15:36.197 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_26 2023-12-10T19:15:36.320 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:15:36.321 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ status_result= "state": "Replaying" 2023-12-10T19:15:36.321 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ + echo "state": "Replaying" 2023-12-10T19:15:36.321 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:grep -i Replaying 2023-12-10T19:15:36.321 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "state": "Replaying" 2023-12-10T19:15:36.324 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=started 2023-12-10T19:15:36.324 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test started = started 2023-12-10T19:15:36.324 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:15:36.324 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_26 2023-12-10T19:15:36.324 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:15:36.325 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:15:36.325 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:36.325 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_26 2023-12-10T19:15:36.325 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = journal ] 2023-12-10T19:15:36.325 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = snapshot ] 2023-12-10T19:15:36.325 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_26 2023-12-10T19:15:36.325 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:15:36.325 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:15:36.325 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:36.325 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_26 2023-12-10T19:15:36.325 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster2-mirror-image_26.status 2023-12-10T19:15:36.325 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2-mirror-image_26.status 2023-12-10T19:15:36.326 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:15:36.326 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_26.status 2023-12-10T19:15:36.326 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-image_26.status 2023-12-10T19:15:36.326 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-image_26.status 2023-12-10T19:15:36.326 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:15:36.326 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_26.status 2023-12-10T19:15:36.326 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mirror_image_snapshot cluster2 mirror image_26 2023-12-10T19:15:36.327 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:15:36.327 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:36.327 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_26 2023-12-10T19:15:36.327 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_26 2023-12-10T19:15:36.767 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Snapshot ID: 169 2023-12-10T19:15:36.780 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_26 /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_26.status 2023-12-10T19:15:36.781 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:15:36.781 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:36.781 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_26 2023-12-10T19:15:36.781 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_26.status 2023-12-10T19:15:36.781 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_26 --format xml 2023-12-10T19:15:36.781 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:15:36.837 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-10T19:15:36.840 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snapshot_id=169 2023-12-10T19:15:36.840 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:15:36.840 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.2 2023-12-10T19:15:37.041 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_26.status 2023-12-10T19:15:37.041 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:37.042 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:37.042 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_26 2023-12-10T19:15:37.042 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_26.status 2023-12-10T19:15:37.042 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2023-12-10T19:15:37.042 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:15:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:15:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:15:37.111 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:15:37.111 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:15:37.111 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:15:37.111 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:15:37.112 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 169 = 2023-12-10T19:15:37.112 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.4 2023-12-10T19:15:37.513 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_26.status 2023-12-10T19:15:37.513 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:37.513 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:37.513 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_26 2023-12-10T19:15:37.513 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_26.status 2023-12-10T19:15:37.513 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2023-12-10T19:15:37.513 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:15:37.623 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:15:37.623 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:15:37.624 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:15:37.625 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:15:37.625 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:15:37.625 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:15:37.625 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 169 = 2023-12-10T19:15:37.625 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.8 2023-12-10T19:15:38.427 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_26.status 2023-12-10T19:15:38.427 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:38.427 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:38.427 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_26 2023-12-10T19:15:38.427 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_26.status 2023-12-10T19:15:38.427 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2023-12-10T19:15:38.427 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:15:38.496 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:15:38.500 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=169 2023-12-10T19:15:38.500 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 169 = 169 2023-12-10T19:15:38.500 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:15:38.500 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snap_present cluster1 mirror image_26 snap 2023-12-10T19:15:38.500 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:38.500 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:38.500 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_26 2023-12-10T19:15:38.500 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:15:38.500 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:15:38.500 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:15:39.501 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 info mirror/image_26@snap 2023-12-10T19:15:39.566 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_26': 2023-12-10T19:15:39.566 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:15:39.566 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:15:39.566 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 2 2023-12-10T19:15:39.566 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 10e887e17739 2023-12-10T19:15:39.566 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.10e887e17739 2023-12-10T19:15:39.566 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:15:39.566 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering, non-primary 2023-12-10T19:15:39.566 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:15:39.566 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:15:39.566 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:07:15 2023 2023-12-10T19:15:39.567 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:07:15 2023 2023-12-10T19:15:39.567 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:07:15 2023 2023-12-10T19:15:39.567 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: protected: False 2023-12-10T19:15:39.567 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:15:39.567 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:15:39.567 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: b6b891c6-1bf3-4053-93b5-065fbe9ec26b 2023-12-10T19:15:39.567 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: false 2023-12-10T19:15:39.570 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:15:39.570 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror image_26 snap 2023-12-10T19:15:39.570 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:39.570 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_26 2023-12-10T19:15:39.570 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:15:39.570 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:15:39.570 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_26.export 2023-12-10T19:15:39.571 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_26.export 2023-12-10T19:15:39.571 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_26.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_26.export 2023-12-10T19:15:39.571 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export image_26@snap /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_26.export 2023-12-10T19:15:39.785 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:15:39.793 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export image_26@snap /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_26.export 2023-12-10T19:15:40.101 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:15:40.118 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_26.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_26.export 2023-12-10T19:15:40.196 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_26.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_26.export 2023-12-10T19:15:40.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:15:40.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_27 2023-12-10T19:15:40.238 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror image_27 snap 2023-12-10T19:15:40.238 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:15:40.238 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:40.238 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_27 2023-12-10T19:15:40.238 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:15:40.238 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create image_27@snap --debug-rbd=20 --debug-journaler=20 2023-12-10T19:15:40.763 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_started cluster1 mirror image_27 2023-12-10T19:15:40.764 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:40.764 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:40.764 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_27 2023-12-10T19:15:40.764 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_state cluster1 mirror image_27 started 2023-12-10T19:15:40.764 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:40.764 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:40.764 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_27 2023-12-10T19:15:40.764 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=started 2023-12-10T19:15:40.764 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:15:40.764 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:15:41.765 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_replay_state cluster1 mirror image_27 started 2023-12-10T19:15:41.765 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:41.765 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:41.765 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_27 2023-12-10T19:15:41.765 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=started 2023-12-10T19:15:41.765 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_result 2023-12-10T19:15:41.765 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=stopped 2023-12-10T19:15:41.766 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_27 2023-12-10T19:15:41.766 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_instance=cluster1 2023-12-10T19:15:41.766 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ + shift 2023-12-10T19:15:41.766 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ greplocal -i cluster=cluster1 state 2023-12-10T19:15:41.766 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:15:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance=cluster1 2023-12-10T19:15:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loop_instance 2023-12-10T19:15:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:15:41.768 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T19:15:41.768 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T19:15:41.768 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_27 2023-12-10T19:15:41.768 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T19:15:41.769 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:15:41.769 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:15:41.769 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T19:15:41.769 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:15:41.769 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:15:41.769 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:15:41.769 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:41.769 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:15:41.769 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:15:41.769 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:15:41.769 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:41.769 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:15:41.769 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:15:41.769 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T19:15:41.769 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T19:15:41.770 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:41.770 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:15:41.770 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T19:15:41.770 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:15:41.770 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:15:41.770 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:15:41.770 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:41.770 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:15:41.770 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:15:41.770 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:15:41.770 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:15:41.770 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:15:41.770 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:15:41.770 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T19:15:41.791 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:15:41.791 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:15:41.791 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:15:41.791 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_27 2023-12-10T19:15:41.916 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:15:41.917 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ status_result= "state": "Replaying" 2023-12-10T19:15:41.917 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo "state": "Replaying" 2023-12-10T19:15:41.917 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep -i Replaying 2023-12-10T19:15:41.918 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "state": "Replaying" 2023-12-10T19:15:41.919 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=started 2023-12-10T19:15:41.919 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test started = started 2023-12-10T19:15:41.919 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:15:41.919 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_27 2023-12-10T19:15:41.919 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:15:41.919 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:15:41.919 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:41.919 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_27 2023-12-10T19:15:41.919 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = journal ] 2023-12-10T19:15:41.919 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = snapshot ] 2023-12-10T19:15:41.919 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_27 2023-12-10T19:15:41.920 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:15:41.920 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:15:41.920 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:41.920 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_27 2023-12-10T19:15:41.920 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster2-mirror-image_27.status 2023-12-10T19:15:41.920 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ + sed -e s|[/ ]|_|g 2023-12-10T19:15:41.920 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:echo cluster2-mirror-image_27.status 2023-12-10T19:15:41.921 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_27.status 2023-12-10T19:15:41.921 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-image_27.status 2023-12-10T19:15:41.921 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-image_27.status 2023-12-10T19:15:41.921 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:15:41.921 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_27.status 2023-12-10T19:15:41.921 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mirror_image_snapshot cluster2 mirror image_27 2023-12-10T19:15:41.921 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:15:41.921 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:41.921 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_27 2023-12-10T19:15:41.922 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_27 2023-12-10T19:15:42.781 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Snapshot ID: 171 2023-12-10T19:15:42.795 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_27 /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_27.status 2023-12-10T19:15:42.795 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:15:42.795 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:42.796 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_27 2023-12-10T19:15:42.796 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_27.status 2023-12-10T19:15:42.796 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_27 --format xml 2023-12-10T19:15:42.796 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:15:42.851 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-10T19:15:42.854 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snapshot_id=171 2023-12-10T19:15:42.854 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:15:42.854 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.2 2023-12-10T19:15:43.055 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_27.status 2023-12-10T19:15:43.056 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:43.056 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:43.056 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_27 2023-12-10T19:15:43.056 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_27.status 2023-12-10T19:15:43.056 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2023-12-10T19:15:43.056 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:15:43.127 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:15:43.130 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=94 2023-12-10T19:15:43.130 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 171 = 94 2023-12-10T19:15:43.130 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.4 2023-12-10T19:15:43.532 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_27.status 2023-12-10T19:15:43.532 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:43.532 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:43.532 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_27 2023-12-10T19:15:43.532 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_27.status 2023-12-10T19:15:43.532 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2023-12-10T19:15:43.532 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:15:43.601 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:15:43.601 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:15:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:15:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:15:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:15:43.606 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:15:43.606 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 171 = 2023-12-10T19:15:43.606 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.8 2023-12-10T19:15:43.865 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7206053897356252) > weight (0.25) 2023-12-10T19:15:43.865 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.44602242440189965) > weight (0.25) 2023-12-10T19:15:43.866 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.6104031077097805) > weight (0.25) 2023-12-10T19:15:43.866 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.6990278419736733) > weight (0.25) 2023-12-10T19:15:43.866 INFO:tasks.rbd_mirror_thrash:waiting for 104 secs before thrashing 2023-12-10T19:15:44.405 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_27.status 2023-12-10T19:15:44.405 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:44.405 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:44.405 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_27 2023-12-10T19:15:44.405 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_27.status 2023-12-10T19:15:44.405 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2023-12-10T19:15:44.405 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:15:44.477 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:15:44.478 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:15:44.480 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:15:44.480 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:15:44.480 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:15:44.480 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:15:44.480 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 171 = 2023-12-10T19:15:44.480 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1.6 2023-12-10T19:15:46.082 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_27.status 2023-12-10T19:15:46.082 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:46.082 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:46.082 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_27 2023-12-10T19:15:46.082 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_27.status 2023-12-10T19:15:46.082 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2023-12-10T19:15:46.082 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:15:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:15:46.155 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=171 2023-12-10T19:15:46.155 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 171 = 171 2023-12-10T19:15:46.155 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:15:46.155 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snap_present cluster1 mirror image_27 snap 2023-12-10T19:15:46.155 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:46.155 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:46.156 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_27 2023-12-10T19:15:46.156 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:15:46.156 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:15:46.156 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:15:47.157 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 info mirror/image_27@snap 2023-12-10T19:15:47.220 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_27': 2023-12-10T19:15:47.220 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:15:47.220 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:15:47.220 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 2 2023-12-10T19:15:47.220 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 128b36b16ea9 2023-12-10T19:15:47.220 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.128b36b16ea9 2023-12-10T19:15:47.221 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:15:47.221 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering, non-primary 2023-12-10T19:15:47.221 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:15:47.221 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:15:47.221 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:07:21 2023 2023-12-10T19:15:47.221 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:07:21 2023 2023-12-10T19:15:47.221 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:07:21 2023 2023-12-10T19:15:47.221 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: protected: False 2023-12-10T19:15:47.221 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:15:47.221 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:15:47.221 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: ef2fa6c2-fc2c-4d0b-9dde-4392f17b097a 2023-12-10T19:15:47.221 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: false 2023-12-10T19:15:47.225 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:15:47.225 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror image_27 snap 2023-12-10T19:15:47.225 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:47.225 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_27 2023-12-10T19:15:47.226 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:15:47.226 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:15:47.226 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_27.export 2023-12-10T19:15:47.226 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_27.export 2023-12-10T19:15:47.226 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_27.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_27.export 2023-12-10T19:15:47.226 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export image_27@snap /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_27.export 2023-12-10T19:15:47.426 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:15:47.435 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export image_27@snap /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_27.export 2023-12-10T19:15:47.710 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:15:47.719 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_27.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_27.export 2023-12-10T19:15:47.793 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_27.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_27.export 2023-12-10T19:15:47.833 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:15:47.834 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_28 2023-12-10T19:15:47.834 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror image_28 snap 2023-12-10T19:15:47.834 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:15:47.834 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:47.834 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_28 2023-12-10T19:15:47.834 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:15:47.834 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create image_28@snap --debug-rbd=20 --debug-journaler=20 2023-12-10T19:15:48.706 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_started cluster1 mirror image_28 2023-12-10T19:15:48.706 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:48.706 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:48.706 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_28 2023-12-10T19:15:48.706 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_state cluster1 mirror image_28 started 2023-12-10T19:15:48.706 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:48.706 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:48.706 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_28 2023-12-10T19:15:48.707 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=started 2023-12-10T19:15:48.707 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:15:48.707 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:15:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_replay_state cluster1 mirror image_28 started 2023-12-10T19:15:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_28 2023-12-10T19:15:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=started 2023-12-10T19:15:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_result 2023-12-10T19:15:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=stopped 2023-12-10T19:15:49.708 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_28 2023-12-10T19:15:49.708 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ + local cluster_instance=cluster1 2023-12-10T19:15:49.708 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grepshift -i 2023-12-10T19:15:49.708 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: state 2023-12-10T19:15:49.708 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:49.708 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance=cluster1 2023-12-10T19:15:49.708 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loop_instance 2023-12-10T19:15:49.708 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:15:49.710 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T19:15:49.710 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T19:15:49.710 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_28 2023-12-10T19:15:49.710 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T19:15:49.710 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:15:49.710 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:15:49.711 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T19:15:49.711 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:15:49.711 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:15:49.711 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:15:49.711 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:49.711 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:15:49.711 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:15:49.711 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:15:49.711 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:49.711 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:15:49.711 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:15:49.711 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T19:15:49.711 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T19:15:49.711 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:49.711 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:15:49.711 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T19:15:49.712 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:15:49.712 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:15:49.712 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:15:49.712 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:49.712 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:15:49.712 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:15:49.712 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:15:49.712 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:15:49.712 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:15:49.712 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:15:49.712 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T19:15:49.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:15:49.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:15:49.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:15:49.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_28 2023-12-10T19:15:49.846 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2023-12-10T19:15:49.846 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:15:49.846 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:15:49.848 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_28 2023-12-10T19:15:49.848 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:1 2023-12-10T19:15:49.848 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:15:49.848 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:15:49.850 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-10T19:15:49.850 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:15:49.850 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:15:49.850 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:15:49.850 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:49.850 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:15:49.850 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:15:49.850 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:15:49.850 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:49.850 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:15:49.850 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:15:49.851 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-10T19:15:49.851 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:1 2023-12-10T19:15:49.851 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:49.851 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:15:49.851 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-10T19:15:49.851 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:15:49.851 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:15:49.851 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:15:49.851 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:49.851 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:15:49.851 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:15:49.851 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:15:49.851 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:15:49.851 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:15:49.851 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:15:49.851 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-10T19:15:49.872 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:15:49.872 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:15:49.872 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:15:49.872 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_28 2023-12-10T19:15:49.991 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:15:49.992 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ status_result= "state": "Replaying" 2023-12-10T19:15:49.992 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ + echo "state": "Replaying" 2023-12-10T19:15:49.992 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:grep -i Replaying 2023-12-10T19:15:49.993 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "state": "Replaying" 2023-12-10T19:15:49.995 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=started 2023-12-10T19:15:49.995 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test started = started 2023-12-10T19:15:49.995 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:15:49.995 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_28 2023-12-10T19:15:49.995 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:15:49.995 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:15:49.995 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:49.995 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_28 2023-12-10T19:15:49.995 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = journal ] 2023-12-10T19:15:49.995 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = snapshot ] 2023-12-10T19:15:49.995 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_28 2023-12-10T19:15:49.995 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:15:49.995 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:15:49.995 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:49.996 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_28 2023-12-10T19:15:49.996 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster2-mirror-image_28.status 2023-12-10T19:15:49.996 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2-mirror-image_28.status 2023-12-10T19:15:49.996 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:15:49.997 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_28.status 2023-12-10T19:15:49.997 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-image_28.status 2023-12-10T19:15:49.997 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-image_28.status 2023-12-10T19:15:49.997 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:15:49.997 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_28.status 2023-12-10T19:15:49.997 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mirror_image_snapshot cluster2 mirror image_28 2023-12-10T19:15:49.998 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:15:49.998 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:49.998 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_28 2023-12-10T19:15:49.998 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_28 2023-12-10T19:15:50.738 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Snapshot ID: 173 2023-12-10T19:15:50.752 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_28 /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_28.status 2023-12-10T19:15:50.752 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:15:50.752 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:50.753 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_28 2023-12-10T19:15:50.753 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_28.status 2023-12-10T19:15:50.753 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_28 --format xml 2023-12-10T19:15:50.753 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:15:50.809 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-10T19:15:50.811 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snapshot_id=173 2023-12-10T19:15:50.811 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:15:50.811 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.2 2023-12-10T19:15:51.014 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_28.status 2023-12-10T19:15:51.014 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:51.014 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:51.014 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_28 2023-12-10T19:15:51.014 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_28.status 2023-12-10T19:15:51.014 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2023-12-10T19:15:51.014 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:15:51.086 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:15:51.089 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=95 2023-12-10T19:15:51.089 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 173 = 95 2023-12-10T19:15:51.089 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.4 2023-12-10T19:15:51.490 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_28.status 2023-12-10T19:15:51.490 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:51.491 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:51.491 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_28 2023-12-10T19:15:51.491 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_28.status 2023-12-10T19:15:51.491 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2023-12-10T19:15:51.491 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:15:51.575 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:15:51.575 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:15:51.578 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:15:51.578 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:15:51.578 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:15:51.578 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:15:51.578 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 173 = 2023-12-10T19:15:51.578 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.8 2023-12-10T19:15:52.379 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_28.status 2023-12-10T19:15:52.380 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:52.380 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:52.380 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_28 2023-12-10T19:15:52.380 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_28.status 2023-12-10T19:15:52.380 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2023-12-10T19:15:52.380 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:15:52.446 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:15:52.446 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:15:52.448 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:15:52.448 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:15:52.449 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:15:52.449 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:15:52.449 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 173 = 2023-12-10T19:15:52.449 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1.6 2023-12-10T19:15:54.050 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_28.status 2023-12-10T19:15:54.050 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:54.050 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:54.050 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_28 2023-12-10T19:15:54.050 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_28.status 2023-12-10T19:15:54.050 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2023-12-10T19:15:54.051 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:15:54.118 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:15:54.120 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=173 2023-12-10T19:15:54.120 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 173 = 173 2023-12-10T19:15:54.121 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:15:54.121 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snap_present cluster1 mirror image_28 snap 2023-12-10T19:15:54.121 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:54.121 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:54.121 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_28 2023-12-10T19:15:54.121 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:15:54.121 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:15:54.121 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:15:55.121 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 info mirror/image_28@snap 2023-12-10T19:15:55.194 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_28': 2023-12-10T19:15:55.194 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:15:55.194 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:15:55.194 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 2 2023-12-10T19:15:55.194 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 129b1967fba7 2023-12-10T19:15:55.194 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.129b1967fba7 2023-12-10T19:15:55.194 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:15:55.194 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering, non-primary 2023-12-10T19:15:55.194 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:15:55.194 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:15:55.194 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:07:32 2023 2023-12-10T19:15:55.195 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:07:32 2023 2023-12-10T19:15:55.195 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:07:32 2023 2023-12-10T19:15:55.195 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: protected: False 2023-12-10T19:15:55.195 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:15:55.195 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:15:55.195 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: 8f8a9063-1808-4763-bb98-81cd23d63bba 2023-12-10T19:15:55.195 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: false 2023-12-10T19:15:55.198 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:15:55.198 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror image_28 snap 2023-12-10T19:15:55.198 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:55.198 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_28 2023-12-10T19:15:55.198 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:15:55.198 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:15:55.198 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_28.export 2023-12-10T19:15:55.198 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_28.export 2023-12-10T19:15:55.198 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_28.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_28.export 2023-12-10T19:15:55.199 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export image_28@snap /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_28.export 2023-12-10T19:15:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:15:55.409 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export image_28@snap /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_28.export 2023-12-10T19:15:55.763 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:15:55.780 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_28.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_28.export 2023-12-10T19:15:55.858 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_28.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_28.export 2023-12-10T19:15:55.898 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:15:55.898 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_29 2023-12-10T19:15:55.898 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror image_29 snap 2023-12-10T19:15:55.898 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:15:55.898 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:55.899 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_29 2023-12-10T19:15:55.899 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:15:55.899 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create image_29@snap --debug-rbd=20 --debug-journaler=20 2023-12-10T19:15:56.388 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_started cluster1 mirror image_29 2023-12-10T19:15:56.388 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:56.389 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:56.389 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_29 2023-12-10T19:15:56.389 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_state cluster1 mirror image_29 started 2023-12-10T19:15:56.389 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:56.389 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:56.389 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_29 2023-12-10T19:15:56.389 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=started 2023-12-10T19:15:56.389 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:15:56.389 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:15:57.390 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_replay_state cluster1 mirror image_29 started 2023-12-10T19:15:57.390 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:57.390 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:15:57.390 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_29 2023-12-10T19:15:57.390 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=started 2023-12-10T19:15:57.390 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_result 2023-12-10T19:15:57.390 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=stopped 2023-12-10T19:15:57.391 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_29 2023-12-10T19:15:57.391 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ + local cluster_instance=cluster1 2023-12-10T19:15:57.391 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:15:57.391 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:grep -i state 2023-12-10T19:15:57.391 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:57.391 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance=cluster1 2023-12-10T19:15:57.391 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loop_instance 2023-12-10T19:15:57.391 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:15:57.391 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T19:15:57.392 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T19:15:57.394 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_29 2023-12-10T19:15:57.394 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T19:15:57.394 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:15:57.394 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:15:57.394 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T19:15:57.394 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:15:57.394 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:15:57.394 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:15:57.394 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:57.394 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:15:57.394 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:15:57.394 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:15:57.394 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:57.395 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:15:57.395 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:15:57.395 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T19:15:57.395 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T19:15:57.395 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:57.395 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:15:57.395 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T19:15:57.395 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:15:57.395 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:15:57.395 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:15:57.395 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:57.395 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:15:57.395 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:15:57.395 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:15:57.395 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:15:57.395 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:15:57.395 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:15:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T19:15:57.418 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:15:57.419 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:15:57.419 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:15:57.419 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_29 2023-12-10T19:15:57.530 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2023-12-10T19:15:57.530 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:15:57.531 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:15:57.533 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_29 2023-12-10T19:15:57.533 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:1 2023-12-10T19:15:57.533 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:15:57.533 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:15:57.533 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-10T19:15:57.533 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:15:57.533 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:15:57.533 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:15:57.533 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:57.533 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:15:57.533 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:15:57.533 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:15:57.533 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:57.533 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:15:57.534 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:15:57.534 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-10T19:15:57.534 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:1 2023-12-10T19:15:57.534 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:57.534 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:15:57.534 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-10T19:15:57.534 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:15:57.534 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:15:57.534 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:15:57.534 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:57.534 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:15:57.534 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:15:57.534 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:15:57.534 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:15:57.534 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:15:57.534 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:15:57.535 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-10T19:15:57.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:15:57.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:15:57.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:15:57.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_29 2023-12-10T19:15:57.671 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2023-12-10T19:15:57.671 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:15:57.671 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:15:57.673 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_29 2023-12-10T19:15:57.673 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:2 2023-12-10T19:15:57.673 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:15:57.673 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:15:57.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-10T19:15:57.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:15:57.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:15:57.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:15:57.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:57.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:15:57.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:15:57.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:15:57.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:57.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:15:57.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:15:57.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-10T19:15:57.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:2 2023-12-10T19:15:57.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:57.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:15:57.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-10T19:15:57.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:15:57.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:15:57.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:15:57.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:57.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:15:57.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:15:57.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:15:57.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:15:57.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:15:57.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:15:57.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-10T19:15:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:15:57.699 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:15:57.699 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:15:57.699 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_29 2023-12-10T19:15:57.800 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2023-12-10T19:15:57.800 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:15:57.801 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:15:57.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_29 2023-12-10T19:15:57.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:3 2023-12-10T19:15:57.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:15:57.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:15:57.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-10T19:15:57.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:3 2023-12-10T19:15:57.805 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:15:57.805 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:15:57.805 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:57.805 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:15:57.805 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 3 = cluster1:3 ] 2023-12-10T19:15:57.805 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:15:57.805 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:57.805 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=3 2023-12-10T19:15:57.805 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:15:57.805 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-10T19:15:57.805 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:3 2023-12-10T19:15:57.805 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:57.805 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:15:57.805 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-10T19:15:57.805 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:3 2023-12-10T19:15:57.805 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:15:57.806 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:15:57.806 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:57.806 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:15:57.806 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 3 = cluster1:3 ] 2023-12-10T19:15:57.806 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:15:57.806 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:15:57.806 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=3 2023-12-10T19:15:57.806 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:15:57.806 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-10T19:15:57.827 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T19:15:57.827 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T19:15:57.827 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T19:15:57.827 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_29 2023-12-10T19:15:57.933 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2023-12-10T19:15:57.933 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:15:57.933 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:15:57.934 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:15:58.936 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T19:15:58.936 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T19:15:58.938 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_29 2023-12-10T19:15:58.938 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T19:15:58.938 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:15:58.938 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:15:58.938 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T19:15:58.938 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:15:58.938 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:15:58.938 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:15:58.938 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:58.938 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:15:58.938 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:15:58.938 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:15:58.938 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:58.939 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:15:58.939 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:15:58.939 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T19:15:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T19:15:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:15:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T19:15:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:15:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:15:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:15:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:15:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:15:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:15:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:15:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:15:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:15:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T19:15:58.964 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:15:58.964 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:15:58.964 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:15:58.965 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_29 2023-12-10T19:15:59.097 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2023-12-10T19:15:59.097 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:15:59.097 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:15:59.099 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_29 2023-12-10T19:15:59.099 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:1 2023-12-10T19:15:59.099 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:15:59.099 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:15:59.099 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-10T19:15:59.099 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:15:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:15:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:15:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:15:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:15:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:15:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:15:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:15:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-10T19:15:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:1 2023-12-10T19:15:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:15:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-10T19:15:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:15:59.101 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:15:59.101 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:15:59.101 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:59.101 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:15:59.101 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:15:59.101 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:15:59.101 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:15:59.101 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:15:59.101 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:15:59.102 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-10T19:15:59.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:15:59.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:15:59.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:15:59.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_29 2023-12-10T19:15:59.229 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2023-12-10T19:15:59.229 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:15:59.229 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:15:59.232 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_29 2023-12-10T19:15:59.232 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:2 2023-12-10T19:15:59.232 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:15:59.232 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:15:59.232 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-10T19:15:59.232 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:15:59.232 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:15:59.232 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:15:59.232 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:59.232 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:15:59.232 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:15:59.232 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:15:59.232 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:59.232 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:15:59.233 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:15:59.233 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-10T19:15:59.233 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:2 2023-12-10T19:15:59.233 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:59.233 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:15:59.233 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-10T19:15:59.233 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:15:59.233 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:15:59.233 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:15:59.233 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:59.233 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:15:59.233 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:15:59.234 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:15:59.234 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:15:59.234 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:15:59.234 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:15:59.234 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-10T19:15:59.251 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:15:59.251 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:15:59.251 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:15:59.251 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_29 2023-12-10T19:15:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2023-12-10T19:15:59.407 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:15:59.407 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:15:59.409 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_29 2023-12-10T19:15:59.409 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:3 2023-12-10T19:15:59.409 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:15:59.409 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:15:59.409 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-10T19:15:59.409 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:3 2023-12-10T19:15:59.409 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:15:59.409 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:15:59.409 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:59.409 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:15:59.409 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 3 = cluster1:3 ] 2023-12-10T19:15:59.409 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:15:59.409 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:59.410 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=3 2023-12-10T19:15:59.410 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:15:59.410 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-10T19:15:59.410 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:3 2023-12-10T19:15:59.410 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:15:59.410 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:15:59.410 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-10T19:15:59.410 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:3 2023-12-10T19:15:59.410 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:15:59.410 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:15:59.410 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:15:59.410 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:15:59.410 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 3 = cluster1:3 ] 2023-12-10T19:15:59.410 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:15:59.410 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:15:59.411 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=3 2023-12-10T19:15:59.411 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:15:59.411 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-10T19:15:59.432 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T19:15:59.432 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T19:15:59.432 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T19:15:59.432 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_29 2023-12-10T19:15:59.541 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2023-12-10T19:15:59.542 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:15:59.542 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:15:59.543 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 2 2023-12-10T19:16:01.545 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T19:16:01.545 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T19:16:01.545 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_29 2023-12-10T19:16:01.546 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T19:16:01.546 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:16:01.546 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:16:01.546 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T19:16:01.546 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:16:01.546 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:16:01.546 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:16:01.546 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:01.546 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:16:01.546 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:16:01.546 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:16:01.546 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:01.546 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:16:01.546 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:16:01.547 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T19:16:01.548 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T19:16:01.548 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:01.548 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:16:01.548 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T19:16:01.548 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:16:01.548 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:16:01.548 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:16:01.548 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:01.548 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:16:01.548 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:16:01.548 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:16:01.548 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:16:01.548 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:16:01.548 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:16:01.548 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T19:16:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:16:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:16:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:16:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_29 2023-12-10T19:16:01.690 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2023-12-10T19:16:01.691 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:16:01.691 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:16:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_29 2023-12-10T19:16:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:1 2023-12-10T19:16:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:16:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:16:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-10T19:16:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:16:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:16:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:16:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:01.693 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:16:01.693 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:16:01.693 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:16:01.693 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:01.693 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:16:01.693 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:16:01.694 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-10T19:16:01.694 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:1 2023-12-10T19:16:01.694 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:01.694 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:16:01.694 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-10T19:16:01.694 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:16:01.694 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:16:01.694 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:16:01.694 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:01.695 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:16:01.695 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:16:01.695 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:16:01.695 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:16:01.695 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:16:01.695 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:16:01.695 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-10T19:16:01.717 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:16:01.717 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:16:01.717 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:16:01.717 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_29 2023-12-10T19:16:01.845 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:16:01.845 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ status_result= "state": "Replaying" 2023-12-10T19:16:01.845 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo "state": "Replaying" 2023-12-10T19:16:01.845 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep -i Replaying 2023-12-10T19:16:01.847 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "state": "Replaying" 2023-12-10T19:16:01.848 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=started 2023-12-10T19:16:01.848 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test started = started 2023-12-10T19:16:01.848 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:16:01.848 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_29 2023-12-10T19:16:01.848 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:16:01.848 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:16:01.848 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:16:01.848 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_29 2023-12-10T19:16:01.848 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = journal ] 2023-12-10T19:16:01.848 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = snapshot ] 2023-12-10T19:16:01.848 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_29 2023-12-10T19:16:01.848 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:16:01.849 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:16:01.849 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:16:01.849 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_29 2023-12-10T19:16:01.849 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster2-mirror-image_29.status 2023-12-10T19:16:01.849 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2-mirror-image_29.status 2023-12-10T19:16:01.849 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:16:01.850 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_29.status 2023-12-10T19:16:01.850 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-image_29.status 2023-12-10T19:16:01.850 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-image_29.status 2023-12-10T19:16:01.850 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:16:01.851 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_29.status 2023-12-10T19:16:01.851 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mirror_image_snapshot cluster2 mirror image_29 2023-12-10T19:16:01.851 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:16:01.851 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:16:01.851 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_29 2023-12-10T19:16:01.851 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_29 2023-12-10T19:16:02.736 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Snapshot ID: 175 2023-12-10T19:16:02.746 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_29 /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_29.status 2023-12-10T19:16:02.746 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:16:02.747 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:16:02.747 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_29 2023-12-10T19:16:02.747 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_29.status 2023-12-10T19:16:02.747 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_29 --format xml 2023-12-10T19:16:02.747 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:16:02.801 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-10T19:16:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snapshot_id=175 2023-12-10T19:16:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:16:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.2 2023-12-10T19:16:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_29.status 2023-12-10T19:16:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:16:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_29 2023-12-10T19:16:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_29.status 2023-12-10T19:16:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2023-12-10T19:16:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:16:03.085 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:16:03.088 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=96 2023-12-10T19:16:03.088 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 175 = 96 2023-12-10T19:16:03.088 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.4 2023-12-10T19:16:03.490 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_29.status 2023-12-10T19:16:03.490 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:03.490 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:16:03.490 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_29 2023-12-10T19:16:03.491 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_29.status 2023-12-10T19:16:03.491 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2023-12-10T19:16:03.491 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:16:03.558 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:16:03.561 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=96 2023-12-10T19:16:03.561 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 175 = 96 2023-12-10T19:16:03.561 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.8 2023-12-10T19:16:04.362 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_29.status 2023-12-10T19:16:04.362 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:04.362 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:16:04.362 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_29 2023-12-10T19:16:04.362 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_29.status 2023-12-10T19:16:04.362 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2023-12-10T19:16:04.363 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:16:04.430 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:16:04.430 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:16:04.432 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:16:04.432 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:16:04.432 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:16:04.433 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:16:04.433 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 175 = 2023-12-10T19:16:04.433 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1.6 2023-12-10T19:16:06.034 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_29.status 2023-12-10T19:16:06.034 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:06.034 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:16:06.034 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_29 2023-12-10T19:16:06.034 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_29.status 2023-12-10T19:16:06.034 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2023-12-10T19:16:06.034 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:16:06.102 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:16:06.105 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=175 2023-12-10T19:16:06.105 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 175 = 175 2023-12-10T19:16:06.105 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:16:06.105 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snap_present cluster1 mirror image_29 snap 2023-12-10T19:16:06.105 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:06.105 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:16:06.105 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_29 2023-12-10T19:16:06.105 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:16:06.105 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:16:06.105 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:16:07.106 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 info mirror/image_29@snap 2023-12-10T19:16:07.168 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_29': 2023-12-10T19:16:07.168 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:16:07.168 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:16:07.169 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 2 2023-12-10T19:16:07.169 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 10c04ae2a87a 2023-12-10T19:16:07.169 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.10c04ae2a87a 2023-12-10T19:16:07.169 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:16:07.169 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering, non-primary 2023-12-10T19:16:07.169 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:16:07.169 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:16:07.170 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:07:42 2023 2023-12-10T19:16:07.170 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:07:42 2023 2023-12-10T19:16:07.170 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:07:42 2023 2023-12-10T19:16:07.170 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: protected: False 2023-12-10T19:16:07.170 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:16:07.170 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:16:07.171 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: dc1ce313-d80e-4cd6-a4dd-74802559481e 2023-12-10T19:16:07.171 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: false 2023-12-10T19:16:07.173 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:16:07.173 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror image_29 snap 2023-12-10T19:16:07.173 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:16:07.173 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_29 2023-12-10T19:16:07.173 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:16:07.173 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:16:07.173 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_29.export 2023-12-10T19:16:07.173 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_29.export 2023-12-10T19:16:07.173 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_29.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_29.export 2023-12-10T19:16:07.173 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export image_29@snap /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_29.export 2023-12-10T19:16:07.383 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:16:07.391 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export image_29@snap /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_29.export 2023-12-10T19:16:07.656 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:16:07.673 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_29.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_29.export 2023-12-10T19:16:07.749 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_29.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_29.export 2023-12-10T19:16:07.789 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:16:07.789 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_30 2023-12-10T19:16:07.789 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror image_30 snap 2023-12-10T19:16:07.790 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:16:07.790 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:16:07.790 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_30 2023-12-10T19:16:07.790 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:16:07.790 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create image_30@snap --debug-rbd=20 --debug-journaler=20 2023-12-10T19:16:08.725 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_started cluster1 mirror image_30 2023-12-10T19:16:08.725 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:08.725 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:16:08.726 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_30 2023-12-10T19:16:08.726 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_state cluster1 mirror image_30 started 2023-12-10T19:16:08.726 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:08.726 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:16:08.726 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_30 2023-12-10T19:16:08.726 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=started 2023-12-10T19:16:08.726 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:16:08.726 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:16:09.726 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_replay_state cluster1 mirror image_30 started 2023-12-10T19:16:09.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:09.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:16:09.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_30 2023-12-10T19:16:09.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=started 2023-12-10T19:16:09.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_result 2023-12-10T19:16:09.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=stopped 2023-12-10T19:16:09.728 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_30 2023-12-10T19:16:09.728 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ + local cluster_instance=cluster1 2023-12-10T19:16:09.728 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:16:09.728 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1grep 2023-12-10T19:16:09.728 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: -i state 2023-12-10T19:16:09.728 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance=cluster1 2023-12-10T19:16:09.728 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loop_instance 2023-12-10T19:16:09.728 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:16:09.728 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T19:16:09.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T19:16:09.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_30 2023-12-10T19:16:09.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T19:16:09.731 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:16:09.731 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:16:09.731 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T19:16:09.731 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:16:09.731 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:16:09.731 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:16:09.731 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:09.731 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:16:09.731 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:16:09.731 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:16:09.731 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:09.731 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:16:09.731 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:16:09.731 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T19:16:09.731 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T19:16:09.731 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:09.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:16:09.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T19:16:09.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:16:09.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:16:09.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:16:09.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:09.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:16:09.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:16:09.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:16:09.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:16:09.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:16:09.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:16:09.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T19:16:09.751 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:16:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:16:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:16:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_30 2023-12-10T19:16:09.867 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2023-12-10T19:16:09.867 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:16:09.867 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:16:09.870 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_30 2023-12-10T19:16:09.870 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:1 2023-12-10T19:16:09.870 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:16:09.871 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:16:09.871 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-10T19:16:09.871 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:16:09.871 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:16:09.871 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:16:09.871 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:09.871 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:16:09.872 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:16:09.872 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:16:09.872 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:09.872 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:16:09.872 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:16:09.872 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-10T19:16:09.872 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:1 2023-12-10T19:16:09.873 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:09.873 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:16:09.873 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-10T19:16:09.874 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:16:09.874 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:16:09.874 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:16:09.874 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:09.874 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:16:09.874 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:16:09.874 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:16:09.878 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:16:09.878 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:16:09.878 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:16:09.878 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-10T19:16:09.893 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:16:09.893 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:16:09.893 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:16:09.893 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_30 2023-12-10T19:16:10.005 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2023-12-10T19:16:10.006 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:16:10.006 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:16:10.007 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_30 2023-12-10T19:16:10.007 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:2 2023-12-10T19:16:10.008 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:16:10.008 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:16:10.008 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-10T19:16:10.008 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:16:10.008 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:16:10.008 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:16:10.008 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:10.008 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:16:10.008 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:16:10.008 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:16:10.008 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:10.008 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:16:10.008 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:16:10.008 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-10T19:16:10.008 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:2 2023-12-10T19:16:10.009 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:10.009 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:16:10.009 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-10T19:16:10.009 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:16:10.009 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:16:10.009 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:16:10.009 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:10.009 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:16:10.009 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:16:10.009 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:16:10.009 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:16:10.009 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:16:10.009 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:16:10.009 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-10T19:16:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:16:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:16:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:16:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_30 2023-12-10T19:16:10.141 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2023-12-10T19:16:10.141 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:16:10.141 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:16:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_30 2023-12-10T19:16:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:3 2023-12-10T19:16:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:16:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:16:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-10T19:16:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:3 2023-12-10T19:16:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:16:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:16:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:16:10.144 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 3 = cluster1:3 ] 2023-12-10T19:16:10.144 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:16:10.144 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:10.144 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=3 2023-12-10T19:16:10.144 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:16:10.144 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-10T19:16:10.144 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:3 2023-12-10T19:16:10.144 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:10.144 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:16:10.144 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-10T19:16:10.144 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:3 2023-12-10T19:16:10.144 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:16:10.144 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:16:10.144 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:10.144 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:16:10.144 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 3 = cluster1:3 ] 2023-12-10T19:16:10.145 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:16:10.145 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:16:10.145 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=3 2023-12-10T19:16:10.145 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:16:10.145 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-10T19:16:10.162 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T19:16:10.162 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T19:16:10.162 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T19:16:10.162 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_30 2023-12-10T19:16:10.303 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:16:10.304 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ status_result= "state": "Replaying" 2023-12-10T19:16:10.305 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo "state": "Replaying" 2023-12-10T19:16:10.305 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep -i Replaying 2023-12-10T19:16:10.305 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "state": "Replaying" 2023-12-10T19:16:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=started 2023-12-10T19:16:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test started = started 2023-12-10T19:16:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:16:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_30 2023-12-10T19:16:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:16:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:16:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:16:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_30 2023-12-10T19:16:10.309 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = journal ] 2023-12-10T19:16:10.309 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = snapshot ] 2023-12-10T19:16:10.309 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_30 2023-12-10T19:16:10.309 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:16:10.309 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:16:10.309 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:16:10.309 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_30 2023-12-10T19:16:10.309 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster2-mirror-image_30.status 2023-12-10T19:16:10.309 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ + echo cluster2-mirror-image_30.status 2023-12-10T19:16:10.309 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:sed -e s|[/ ]|_|g 2023-12-10T19:16:10.309 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_30.status 2023-12-10T19:16:10.309 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-image_30.status 2023-12-10T19:16:10.309 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-image_30.status 2023-12-10T19:16:10.309 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:16:10.310 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_30.status 2023-12-10T19:16:10.310 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mirror_image_snapshot cluster2 mirror image_30 2023-12-10T19:16:10.310 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:16:10.310 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:16:10.310 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_30 2023-12-10T19:16:10.310 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_30 2023-12-10T19:16:10.742 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Snapshot ID: 177 2023-12-10T19:16:10.758 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_30 /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_30.status 2023-12-10T19:16:10.758 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:16:10.758 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:16:10.758 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_30 2023-12-10T19:16:10.759 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_30.status 2023-12-10T19:16:10.759 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_30 --format xml 2023-12-10T19:16:10.759 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:16:10.813 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-10T19:16:10.816 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snapshot_id=177 2023-12-10T19:16:10.816 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:16:10.816 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.2 2023-12-10T19:16:11.017 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_30.status 2023-12-10T19:16:11.017 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:11.017 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:16:11.017 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_30 2023-12-10T19:16:11.017 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_30.status 2023-12-10T19:16:11.017 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2023-12-10T19:16:11.017 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:16:11.087 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:16:11.089 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=97 2023-12-10T19:16:11.089 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 177 = 97 2023-12-10T19:16:11.089 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.4 2023-12-10T19:16:11.491 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_30.status 2023-12-10T19:16:11.491 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:11.491 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:16:11.491 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_30 2023-12-10T19:16:11.491 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_30.status 2023-12-10T19:16:11.491 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2023-12-10T19:16:11.491 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:16:11.562 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:16:11.564 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=97 2023-12-10T19:16:11.564 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 177 = 97 2023-12-10T19:16:11.564 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.8 2023-12-10T19:16:12.365 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_30.status 2023-12-10T19:16:12.366 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:12.366 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:16:12.366 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_30 2023-12-10T19:16:12.366 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_30.status 2023-12-10T19:16:12.366 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2023-12-10T19:16:12.366 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:16:12.432 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:16:12.433 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:16:12.435 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:16:12.435 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:16:12.435 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:16:12.435 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:16:12.435 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 177 = 2023-12-10T19:16:12.435 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1.6 2023-12-10T19:16:14.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_30.status 2023-12-10T19:16:14.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:14.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:16:14.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_30 2023-12-10T19:16:14.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_30.status 2023-12-10T19:16:14.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2023-12-10T19:16:14.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:16:14.105 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:16:14.108 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=177 2023-12-10T19:16:14.108 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 177 = 177 2023-12-10T19:16:14.108 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:16:14.109 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snap_present cluster1 mirror image_30 snap 2023-12-10T19:16:14.109 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:14.109 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:16:14.109 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_30 2023-12-10T19:16:14.109 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:16:14.109 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:16:14.109 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:16:15.108 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 info mirror/image_30@snap 2023-12-10T19:16:15.169 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_30': 2023-12-10T19:16:15.169 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:16:15.169 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:16:15.169 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 2 2023-12-10T19:16:15.169 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 10e87c5f2822 2023-12-10T19:16:15.169 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.10e87c5f2822 2023-12-10T19:16:15.169 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:16:15.169 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering, non-primary 2023-12-10T19:16:15.169 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:16:15.169 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:16:15.169 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:07:56 2023 2023-12-10T19:16:15.169 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:07:56 2023 2023-12-10T19:16:15.169 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:07:56 2023 2023-12-10T19:16:15.170 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: protected: False 2023-12-10T19:16:15.170 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:16:15.170 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:16:15.170 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: 4d4eb965-6ed2-4ccb-a76d-8e89ea006260 2023-12-10T19:16:15.170 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: false 2023-12-10T19:16:15.173 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:16:15.173 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror image_30 snap 2023-12-10T19:16:15.173 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:16:15.173 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_30 2023-12-10T19:16:15.173 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:16:15.173 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:16:15.173 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_30.export 2023-12-10T19:16:15.174 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_30.export 2023-12-10T19:16:15.174 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_30.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_30.export 2023-12-10T19:16:15.174 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export image_30@snap /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_30.export 2023-12-10T19:16:15.390 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:16:15.399 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export image_30@snap /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_30.export 2023-12-10T19:16:15.728 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:16:15.739 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_30.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_30.export 2023-12-10T19:16:15.817 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_30.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_30.export 2023-12-10T19:16:15.857 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:16:15.857 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_31 2023-12-10T19:16:15.857 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror image_31 snap 2023-12-10T19:16:15.857 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:16:15.857 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:16:15.858 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_31 2023-12-10T19:16:15.858 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:16:15.858 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create image_31@snap --debug-rbd=20 --debug-journaler=20 2023-12-10T19:16:16.390 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_started cluster1 mirror image_31 2023-12-10T19:16:16.390 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:16.390 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:16:16.390 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_31 2023-12-10T19:16:16.390 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_state cluster1 mirror image_31 started 2023-12-10T19:16:16.390 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:16.390 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:16:16.390 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_31 2023-12-10T19:16:16.390 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=started 2023-12-10T19:16:16.391 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:16:16.391 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:16:17.391 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_replay_state cluster1 mirror image_31 started 2023-12-10T19:16:17.392 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:17.392 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:16:17.392 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_31 2023-12-10T19:16:17.392 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=started 2023-12-10T19:16:17.392 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_result 2023-12-10T19:16:17.392 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=stopped 2023-12-10T19:16:17.393 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_31 2023-12-10T19:16:17.393 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_instance=cluster1 2023-12-10T19:16:17.393 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:16:17.393 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ + local cluster=cluster1 2023-12-10T19:16:17.393 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance=cluster1 2023-12-10T19:16:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ localgrep loop_instance -i 2023-12-10T19:16:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: state 2023-12-10T19:16:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:16:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T19:16:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T19:16:17.395 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_31 2023-12-10T19:16:17.395 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T19:16:17.396 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:16:17.396 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:16:17.396 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T19:16:17.396 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:16:17.396 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:16:17.396 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:16:17.396 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:17.396 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:16:17.396 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:16:17.396 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:16:17.396 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:17.397 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:16:17.397 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:16:17.397 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T19:16:17.397 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T19:16:17.397 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:17.397 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:16:17.397 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T19:16:17.397 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:16:17.397 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:16:17.397 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:16:17.397 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:17.397 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:16:17.397 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:16:17.398 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:16:17.398 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:16:17.398 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:16:17.398 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:16:17.398 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T19:16:17.420 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:16:17.420 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:16:17.421 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:16:17.421 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_31 2023-12-10T19:16:17.535 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2023-12-10T19:16:17.536 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:16:17.536 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:16:17.538 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_31 2023-12-10T19:16:17.538 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:1 2023-12-10T19:16:17.538 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:16:17.538 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:16:17.538 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-10T19:16:17.538 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:16:17.538 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:16:17.538 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:16:17.538 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:17.539 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:16:17.539 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:16:17.539 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:16:17.539 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:17.539 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:16:17.539 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:16:17.539 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-10T19:16:17.539 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:1 2023-12-10T19:16:17.539 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:17.539 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:16:17.539 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-10T19:16:17.539 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:16:17.539 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:16:17.540 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:16:17.540 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:17.540 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:16:17.540 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:16:17.540 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:16:17.540 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:16:17.540 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:16:17.540 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:16:17.540 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-10T19:16:17.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:16:17.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:16:17.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:16:17.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_31 2023-12-10T19:16:17.680 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2023-12-10T19:16:17.680 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:16:17.680 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:16:17.684 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_31 2023-12-10T19:16:17.684 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:2 2023-12-10T19:16:17.684 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:16:17.684 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:16:17.684 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-10T19:16:17.684 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:16:17.684 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:16:17.684 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:16:17.684 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:17.684 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:16:17.684 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:16:17.684 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:16:17.684 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:17.685 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:16:17.685 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:16:17.685 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-10T19:16:17.685 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:2 2023-12-10T19:16:17.685 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:17.685 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:16:17.685 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-10T19:16:17.685 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:16:17.685 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:16:17.685 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:16:17.685 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:17.685 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:16:17.685 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:16:17.685 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:16:17.685 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:16:17.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:16:17.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:16:17.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-10T19:16:17.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:16:17.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:16:17.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:16:17.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_31 2023-12-10T19:16:17.825 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:16:17.826 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ status_result= "state": "Replaying" 2023-12-10T19:16:17.826 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo "state": "Replaying" 2023-12-10T19:16:17.826 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep -i Replaying 2023-12-10T19:16:17.827 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "state": "Replaying" 2023-12-10T19:16:17.830 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=started 2023-12-10T19:16:17.830 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test started = started 2023-12-10T19:16:17.830 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:16:17.830 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_31 2023-12-10T19:16:17.830 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:16:17.830 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:16:17.830 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:16:17.830 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_31 2023-12-10T19:16:17.831 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = journal ] 2023-12-10T19:16:17.831 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = snapshot ] 2023-12-10T19:16:17.831 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_31 2023-12-10T19:16:17.831 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:16:17.831 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:16:17.831 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:16:17.831 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_31 2023-12-10T19:16:17.831 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster2-mirror-image_31.status 2023-12-10T19:16:17.831 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2-mirror-image_31.status 2023-12-10T19:16:17.831 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:16:17.831 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_31.status 2023-12-10T19:16:17.831 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-image_31.status 2023-12-10T19:16:17.832 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-image_31.status 2023-12-10T19:16:17.832 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:16:17.832 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_31.status 2023-12-10T19:16:17.832 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mirror_image_snapshot cluster2 mirror image_31 2023-12-10T19:16:17.832 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:16:17.832 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:16:17.832 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_31 2023-12-10T19:16:17.832 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_31 2023-12-10T19:16:18.414 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Snapshot ID: 179 2023-12-10T19:16:18.427 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_31 /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_31.status 2023-12-10T19:16:18.427 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:16:18.428 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:16:18.428 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_31 2023-12-10T19:16:18.428 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_31.status 2023-12-10T19:16:18.428 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_31 --format xml 2023-12-10T19:16:18.428 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:16:18.483 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-10T19:16:18.485 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snapshot_id=179 2023-12-10T19:16:18.486 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:16:18.486 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.2 2023-12-10T19:16:18.687 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_31.status 2023-12-10T19:16:18.687 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:18.687 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:16:18.687 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_31 2023-12-10T19:16:18.687 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_31.status 2023-12-10T19:16:18.687 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2023-12-10T19:16:18.687 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:16:18.754 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:16:18.754 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:16:18.757 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:16:18.757 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:16:18.757 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:16:18.757 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:16:18.757 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 179 = 2023-12-10T19:16:18.757 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.4 2023-12-10T19:16:19.158 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_31.status 2023-12-10T19:16:19.159 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:19.159 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:16:19.159 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_31 2023-12-10T19:16:19.159 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_31.status 2023-12-10T19:16:19.159 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2023-12-10T19:16:19.159 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:16:19.228 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:16:19.228 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:16:19.231 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:16:19.231 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:16:19.231 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:16:19.232 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:16:19.232 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 179 = 2023-12-10T19:16:19.232 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.8 2023-12-10T19:16:20.033 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_31.status 2023-12-10T19:16:20.033 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:20.033 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:16:20.033 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_31 2023-12-10T19:16:20.033 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_31.status 2023-12-10T19:16:20.033 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2023-12-10T19:16:20.033 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:16:20.103 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:16:20.106 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=179 2023-12-10T19:16:20.106 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 179 = 179 2023-12-10T19:16:20.106 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:16:20.106 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snap_present cluster1 mirror image_31 snap 2023-12-10T19:16:20.106 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:20.106 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:16:20.106 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_31 2023-12-10T19:16:20.106 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:16:20.107 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:16:20.107 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:16:21.107 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 info mirror/image_31@snap 2023-12-10T19:16:21.172 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_31': 2023-12-10T19:16:21.172 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:16:21.172 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:16:21.172 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 2 2023-12-10T19:16:21.172 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 128bbceb9b69 2023-12-10T19:16:21.172 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.128bbceb9b69 2023-12-10T19:16:21.172 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:16:21.172 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering, non-primary 2023-12-10T19:16:21.172 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:16:21.173 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:16:21.173 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:08:07 2023 2023-12-10T19:16:21.173 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:08:07 2023 2023-12-10T19:16:21.173 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:08:07 2023 2023-12-10T19:16:21.173 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: protected: False 2023-12-10T19:16:21.173 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:16:21.173 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:16:21.173 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: 899d2f20-51d9-4c49-ba24-b07293c27322 2023-12-10T19:16:21.173 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: false 2023-12-10T19:16:21.176 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:16:21.176 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror image_31 snap 2023-12-10T19:16:21.176 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:16:21.176 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_31 2023-12-10T19:16:21.177 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:16:21.177 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:16:21.177 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_31.export 2023-12-10T19:16:21.177 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_31.export 2023-12-10T19:16:21.177 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_31.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_31.export 2023-12-10T19:16:21.177 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export image_31@snap /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_31.export 2023-12-10T19:16:21.400 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:16:21.410 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export image_31@snap /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_31.export 2023-12-10T19:16:21.958 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:16:21.975 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_31.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_31.export 2023-12-10T19:16:22.052 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_31.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_31.export 2023-12-10T19:16:22.093 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:16:22.093 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_32 2023-12-10T19:16:22.093 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror image_32 snap 2023-12-10T19:16:22.093 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:16:22.093 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:16:22.093 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_32 2023-12-10T19:16:22.093 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:16:22.093 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create image_32@snap --debug-rbd=20 --debug-journaler=20 2023-12-10T19:16:22.724 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_started cluster1 mirror image_32 2023-12-10T19:16:22.724 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:22.724 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:16:22.725 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_32 2023-12-10T19:16:22.725 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_state cluster1 mirror image_32 started 2023-12-10T19:16:22.725 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:22.725 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:16:22.725 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_32 2023-12-10T19:16:22.725 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=started 2023-12-10T19:16:22.725 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:16:22.725 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:16:23.725 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_replay_state cluster1 mirror image_32 started 2023-12-10T19:16:23.726 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:23.726 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:16:23.726 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_32 2023-12-10T19:16:23.726 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=started 2023-12-10T19:16:23.726 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_result 2023-12-10T19:16:23.726 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=stopped 2023-12-10T19:16:23.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_32 2023-12-10T19:16:23.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_instance=cluster1 2023-12-10T19:16:23.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:16:23.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ + local cluster=cluster1 2023-12-10T19:16:23.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance=cluster1 2023-12-10T19:16:23.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loop_instance 2023-12-10T19:16:23.728 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:grep -i+ state 2023-12-10T19:16:23.728 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:sleep 0 2023-12-10T19:16:23.728 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T19:16:23.728 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T19:16:23.729 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_32 2023-12-10T19:16:23.729 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T19:16:23.729 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:16:23.729 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:16:23.729 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T19:16:23.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:16:23.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:16:23.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:16:23.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:23.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:16:23.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:16:23.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:16:23.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:23.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:16:23.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:16:23.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T19:16:23.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T19:16:23.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:23.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:16:23.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T19:16:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:16:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:16:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:16:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:16:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:16:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:16:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:16:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:16:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:16:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T19:16:23.750 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:16:23.750 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:16:23.750 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:16:23.750 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_32 2023-12-10T19:16:23.863 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2023-12-10T19:16:23.864 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:16:23.864 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:16:23.866 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_32 2023-12-10T19:16:23.866 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:1 2023-12-10T19:16:23.866 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:16:23.866 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:16:23.866 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-10T19:16:23.866 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:16:23.866 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:16:23.867 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:16:23.867 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:23.867 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:16:23.867 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:16:23.867 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:16:23.867 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:23.867 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:16:23.867 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:16:23.867 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-10T19:16:23.867 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:1 2023-12-10T19:16:23.867 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:23.867 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:16:23.867 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-10T19:16:23.867 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:16:23.867 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:16:23.868 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:16:23.868 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:23.868 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:16:23.868 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:16:23.868 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:16:23.868 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:16:23.868 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:16:23.868 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:16:23.868 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-10T19:16:23.890 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:16:23.890 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:16:23.890 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:16:23.890 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_32 2023-12-10T19:16:24.011 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:16:24.012 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ status_result= "state": "Replaying" 2023-12-10T19:16:24.012 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ + echo "state": "Replaying" 2023-12-10T19:16:24.012 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:grep -i Replaying 2023-12-10T19:16:24.014 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "state": "Replaying" 2023-12-10T19:16:24.016 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=started 2023-12-10T19:16:24.016 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test started = started 2023-12-10T19:16:24.016 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:16:24.016 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_32 2023-12-10T19:16:24.016 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:16:24.016 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:16:24.016 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:16:24.017 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_32 2023-12-10T19:16:24.017 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = journal ] 2023-12-10T19:16:24.017 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = snapshot ] 2023-12-10T19:16:24.017 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_32 2023-12-10T19:16:24.017 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:16:24.017 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:16:24.017 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:16:24.017 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_32 2023-12-10T19:16:24.017 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster2-mirror-image_32.status 2023-12-10T19:16:24.017 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ + echo cluster2-mirror-image_32.status 2023-12-10T19:16:24.017 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:sed -e s|[/ ]|_|g 2023-12-10T19:16:24.017 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_32.status 2023-12-10T19:16:24.017 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-image_32.status 2023-12-10T19:16:24.017 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-image_32.status 2023-12-10T19:16:24.017 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:16:24.018 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_32.status 2023-12-10T19:16:24.018 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mirror_image_snapshot cluster2 mirror image_32 2023-12-10T19:16:24.018 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:16:24.019 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:16:24.019 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_32 2023-12-10T19:16:24.019 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_32 2023-12-10T19:16:24.954 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Snapshot ID: 181 2023-12-10T19:16:24.965 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_32 /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_32.status 2023-12-10T19:16:24.965 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:16:24.966 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:16:24.966 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_32 2023-12-10T19:16:24.966 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_32.status 2023-12-10T19:16:24.966 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_32 --format xml 2023-12-10T19:16:24.966 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:16:25.019 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-10T19:16:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snapshot_id=181 2023-12-10T19:16:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:16:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.2 2023-12-10T19:16:25.223 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_32.status 2023-12-10T19:16:25.224 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:25.224 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:16:25.224 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_32 2023-12-10T19:16:25.224 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_32.status 2023-12-10T19:16:25.224 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2023-12-10T19:16:25.224 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:16:25.292 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:16:25.294 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=99 2023-12-10T19:16:25.294 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 181 = 99 2023-12-10T19:16:25.294 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.4 2023-12-10T19:16:25.696 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_32.status 2023-12-10T19:16:25.696 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:25.696 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:16:25.696 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_32 2023-12-10T19:16:25.696 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_32.status 2023-12-10T19:16:25.696 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2023-12-10T19:16:25.696 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:16:25.768 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:16:25.768 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:16:25.771 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:16:25.771 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:16:25.771 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:16:25.771 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:16:25.772 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 181 = 2023-12-10T19:16:25.772 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.8 2023-12-10T19:16:26.573 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_32.status 2023-12-10T19:16:26.573 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:26.573 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:16:26.573 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_32 2023-12-10T19:16:26.573 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_32.status 2023-12-10T19:16:26.573 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2023-12-10T19:16:26.573 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:16:26.641 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:16:26.641 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:16:26.643 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:16:26.643 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:16:26.643 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:16:26.643 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:16:26.643 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 181 = 2023-12-10T19:16:26.643 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1.6 2023-12-10T19:16:28.244 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_32.status 2023-12-10T19:16:28.245 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:28.245 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:16:28.245 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_32 2023-12-10T19:16:28.245 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_32.status 2023-12-10T19:16:28.245 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2023-12-10T19:16:28.245 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:16:28.314 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:16:28.317 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=181 2023-12-10T19:16:28.317 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 181 = 181 2023-12-10T19:16:28.317 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:16:28.317 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snap_present cluster1 mirror image_32 snap 2023-12-10T19:16:28.317 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:28.317 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:16:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_32 2023-12-10T19:16:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:16:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:16:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:16:29.318 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 info mirror/image_32@snap 2023-12-10T19:16:29.379 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_32': 2023-12-10T19:16:29.380 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:16:29.380 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:16:29.380 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 2 2023-12-10T19:16:29.380 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 129b5e4e67ee 2023-12-10T19:16:29.380 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.129b5e4e67ee 2023-12-10T19:16:29.380 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:16:29.380 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering, non-primary 2023-12-10T19:16:29.380 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:16:29.380 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:16:29.380 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:08:14 2023 2023-12-10T19:16:29.380 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:08:14 2023 2023-12-10T19:16:29.380 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:08:14 2023 2023-12-10T19:16:29.380 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: protected: False 2023-12-10T19:16:29.381 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:16:29.381 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:16:29.381 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: f5919ca9-8590-41d3-8074-153a93205199 2023-12-10T19:16:29.381 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: false 2023-12-10T19:16:29.384 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:16:29.384 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror image_32 snap 2023-12-10T19:16:29.384 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:16:29.384 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_32 2023-12-10T19:16:29.385 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:16:29.385 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:16:29.385 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_32.export 2023-12-10T19:16:29.385 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_32.export 2023-12-10T19:16:29.385 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_32.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_32.export 2023-12-10T19:16:29.385 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export image_32@snap /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_32.export 2023-12-10T19:16:29.595 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:16:29.602 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export image_32@snap /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_32.export 2023-12-10T19:16:29.905 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:16:29.920 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_32.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_32.export 2023-12-10T19:16:29.998 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_32.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_32.export 2023-12-10T19:16:30.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:16:30.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_33 2023-12-10T19:16:30.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror image_33 snap 2023-12-10T19:16:30.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:16:30.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:16:30.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_33 2023-12-10T19:16:30.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:16:30.038 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create image_33@snap --debug-rbd=20 --debug-journaler=20 2023-12-10T19:16:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_started cluster1 mirror image_33 2023-12-10T19:16:30.749 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:30.750 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:16:30.750 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_33 2023-12-10T19:16:30.750 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_state cluster1 mirror image_33 started 2023-12-10T19:16:30.751 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:30.751 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:16:30.751 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_33 2023-12-10T19:16:30.751 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=started 2023-12-10T19:16:30.751 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:16:30.752 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:16:31.745 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_replay_state cluster1 mirror image_33 started 2023-12-10T19:16:31.745 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:31.745 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:16:31.745 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_33 2023-12-10T19:16:31.745 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=started 2023-12-10T19:16:31.745 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_result 2023-12-10T19:16:31.745 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=stopped 2023-12-10T19:16:31.746 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemons cluster1+ rbd mirror status mirror/image_33 2023-12-10T19:16:31.746 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:grep -i state 2023-12-10T19:16:31.746 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_instance=cluster1 2023-12-10T19:16:31.746 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:16:31.746 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:31.746 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance=cluster1 2023-12-10T19:16:31.746 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loop_instance 2023-12-10T19:16:31.746 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:16:31.747 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T19:16:31.747 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T19:16:31.747 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_33 2023-12-10T19:16:31.747 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T19:16:31.747 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:16:31.747 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:16:31.748 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T19:16:31.748 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:16:31.748 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:16:31.748 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:16:31.748 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:31.748 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:16:31.748 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:16:31.748 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:16:31.748 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:31.748 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:16:31.748 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:16:31.749 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T19:16:31.749 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T19:16:31.749 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:31.749 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:16:31.749 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T19:16:31.749 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:16:31.749 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:16:31.750 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:16:31.750 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:31.750 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:16:31.750 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:16:31.750 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:16:31.750 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:16:31.750 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:16:31.750 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:16:31.750 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T19:16:31.771 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:16:31.771 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:16:31.771 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:16:31.771 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_33 2023-12-10T19:16:31.883 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2023-12-10T19:16:31.883 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:16:31.883 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:16:31.886 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_33 2023-12-10T19:16:31.886 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:1 2023-12-10T19:16:31.886 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:16:31.886 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:16:31.887 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-10T19:16:31.887 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:16:31.887 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:16:31.887 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:16:31.887 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:31.887 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:16:31.887 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:16:31.887 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:16:31.887 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:31.887 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:16:31.887 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:16:31.887 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-10T19:16:31.887 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:1 2023-12-10T19:16:31.887 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:31.887 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:16:31.888 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-10T19:16:31.888 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:16:31.888 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:16:31.888 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:16:31.888 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:31.888 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:16:31.888 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:16:31.888 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:16:31.888 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:16:31.888 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:16:31.888 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:16:31.888 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-10T19:16:31.910 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:16:31.910 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:16:31.910 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:16:31.910 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_33 2023-12-10T19:16:32.035 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2023-12-10T19:16:32.036 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:16:32.036 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:16:32.036 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_33 2023-12-10T19:16:32.036 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:2 2023-12-10T19:16:32.036 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:16:32.036 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:16:32.036 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-10T19:16:32.036 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:16:32.036 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:16:32.036 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:16:32.036 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:32.036 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:16:32.036 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:16:32.036 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:16:32.036 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:32.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:16:32.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:16:32.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-10T19:16:32.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:2 2023-12-10T19:16:32.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:32.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:16:32.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-10T19:16:32.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:16:32.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:16:32.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:16:32.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:32.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:16:32.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:16:32.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:16:32.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:16:32.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:16:32.038 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:16:32.038 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-10T19:16:32.054 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:16:32.054 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:16:32.055 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:16:32.055 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_33 2023-12-10T19:16:32.173 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:16:32.174 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ status_result= "state": "Replaying" 2023-12-10T19:16:32.174 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo "state": "Replaying" 2023-12-10T19:16:32.174 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep -i Replaying 2023-12-10T19:16:32.175 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "state": "Replaying" 2023-12-10T19:16:32.176 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=started 2023-12-10T19:16:32.176 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test started = started 2023-12-10T19:16:32.177 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:16:32.177 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_33 2023-12-10T19:16:32.177 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:16:32.177 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:16:32.177 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:16:32.177 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_33 2023-12-10T19:16:32.177 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = journal ] 2023-12-10T19:16:32.177 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = snapshot ] 2023-12-10T19:16:32.177 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_33 2023-12-10T19:16:32.177 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:16:32.177 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:16:32.177 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:16:32.177 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_33 2023-12-10T19:16:32.178 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster2-mirror-image_33.status 2023-12-10T19:16:32.178 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2-mirror-image_33.status 2023-12-10T19:16:32.178 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:16:32.179 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_33.status 2023-12-10T19:16:32.179 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-image_33.status 2023-12-10T19:16:32.179 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-image_33.status 2023-12-10T19:16:32.179 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:16:32.179 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_33.status 2023-12-10T19:16:32.179 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mirror_image_snapshot cluster2 mirror image_33 2023-12-10T19:16:32.179 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:16:32.180 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:16:32.180 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_33 2023-12-10T19:16:32.180 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_33 2023-12-10T19:16:32.748 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Snapshot ID: 183 2023-12-10T19:16:32.762 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_33 /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_33.status 2023-12-10T19:16:32.762 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:16:32.763 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:16:32.763 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_33 2023-12-10T19:16:32.763 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_33.status 2023-12-10T19:16:32.763 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_33 --format xml 2023-12-10T19:16:32.763 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:16:32.817 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-10T19:16:32.819 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snapshot_id=183 2023-12-10T19:16:32.820 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:16:32.820 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.2 2023-12-10T19:16:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_33.status 2023-12-10T19:16:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:16:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_33 2023-12-10T19:16:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_33.status 2023-12-10T19:16:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2023-12-10T19:16:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:16:33.107 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:16:33.109 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=100 2023-12-10T19:16:33.109 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 183 = 100 2023-12-10T19:16:33.110 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.4 2023-12-10T19:16:33.511 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_33.status 2023-12-10T19:16:33.511 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:33.511 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:16:33.511 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_33 2023-12-10T19:16:33.511 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_33.status 2023-12-10T19:16:33.511 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2023-12-10T19:16:33.512 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:16:33.582 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:16:33.585 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=100 2023-12-10T19:16:33.585 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 183 = 100 2023-12-10T19:16:33.585 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.8 2023-12-10T19:16:33.873 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-12-10T19:16:33.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-12-10T19:16:33.874 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2023-12-10T19:16:33.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr:2023-12-10T19:16:33.883+0000 7f925ba77640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 54615) UID: 1000 2023-12-10T19:16:34.387 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_33.status 2023-12-10T19:16:34.387 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:34.387 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:16:34.387 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_33 2023-12-10T19:16:34.387 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_33.status 2023-12-10T19:16:34.387 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2023-12-10T19:16:34.387 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:16:34.453 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:16:34.453 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:16:34.455 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:16:34.456 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:16:34.456 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:16:34.456 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:16:34.456 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 183 = 2023-12-10T19:16:34.456 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1.6 2023-12-10T19:16:35.131 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi076.stderr:2023-12-10T19:16:35.139+0000 7f3eb9254640 -1 rbd::mirror::ImageReplayer: 0x55b33c330780 [3/472ccd57-dfb6-4e9c-bb2d-657bb53c8829] start: already running 2023-12-10T19:16:35.131 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi076.stderr:2023-12-10T19:16:35.139+0000 7f3eb8a53640 -1 rbd::mirror::ImageReplayer: 0x55b33c330c80 [3/a4d8c861-aa4e-4c85-9970-1f362ca1c6b6] start: already running 2023-12-10T19:16:35.133 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi076.stderr:2023-12-10T19:16:35.139+0000 7f890c64a640 -1 rbd::mirror::ImageReplayer: 0x55cbc4794000 [3/5f3f027e-9f9a-463e-be32-1e7c54f42e4e] start: already running 2023-12-10T19:16:35.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi076.stderr:2023-12-10T19:16:35.143+0000 7f88fee2f640 -1 rbd::mirror::ImageReplayer: 0x55cbc34cb680 [3/013b5e35-24cb-486f-93bf-0f941ebb1c1d] start: already running 2023-12-10T19:16:35.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi076.stderr:2023-12-10T19:16:35.143+0000 7f3f0da20640 -1 rbd::mirror::ImageReplayer: 0x5620e4937900 [3/509368f6-035e-4be6-926a-dc4bce662e88] start: already running 2023-12-10T19:16:35.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi076.stderr:2023-12-10T19:16:35.143+0000 7f3f0da20640 -1 rbd::mirror::ImageReplayer: 0x5620e4936f00 [3/3140c852-fad6-4a00-a40c-46869faa3ea1] start: already running 2023-12-10T19:16:35.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi076.stderr:2023-12-10T19:16:35.143+0000 7f3eaba39640 -1 rbd::mirror::ImageReplayer: 0x55b33c330f00 [3/dc1ce313-d80e-4cd6-a4dd-74802559481e] start: already running 2023-12-10T19:16:35.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi076.stderr:2023-12-10T19:16:35.143+0000 7f3eaba39640 -1 rbd::mirror::ImageReplayer: 0x55b33ae5af00 [3/f24c77de-98cd-45a4-9355-6e2e919e8517] start: already running 2023-12-10T19:16:35.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi076.stderr:2023-12-10T19:16:35.143+0000 7f3eaba39640 -1 rbd::mirror::ImageReplayer: 0x55b33ae5bb80 [3/35463a96-219e-457b-aaa2-28a56599112e] start: already running 2023-12-10T19:16:35.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi076.stderr:2023-12-10T19:16:35.143+0000 7f3eaba39640 -1 rbd::mirror::ImageReplayer: 0x55b33ae5a780 [3/8f8a9063-1808-4763-bb98-81cd23d63bba] start: already running 2023-12-10T19:16:35.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi076.stderr:2023-12-10T19:16:35.143+0000 7f3eaba39640 -1 rbd::mirror::ImageReplayer: 0x55b33ae5aa00 [3/f2b859e8-6dbd-4781-91cf-11df1eb8a7a0] start: already running 2023-12-10T19:16:35.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi076.stderr:2023-12-10T19:16:35.143+0000 7f3eaba39640 -1 rbd::mirror::ImageReplayer: 0x55b33c330a00 [3/5e79b28d-acd2-4fbc-9a84-6c7b4c7dc476] start: already running 2023-12-10T19:16:35.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi076.stderr:2023-12-10T19:16:35.143+0000 7f3eaba39640 -1 rbd::mirror::ImageReplayer: 0x55b33ae5b900 [3/f5919ca9-8590-41d3-8074-153a93205199] start: already running 2023-12-10T19:16:35.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi076.stderr:2023-12-10T19:16:35.143+0000 7f3f00205640 -1 rbd::mirror::ImageReplayer: 0x5620e4937400 [3/40a91520-4e7a-4ba5-aaf0-64312a2dcb5a] start: already running 2023-12-10T19:16:35.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi076.stderr:2023-12-10T19:16:35.143+0000 7f3eaba39640 -1 rbd::mirror::ImageReplayer: 0x55b33ae5b180 [3/935218e9-f0e7-416b-a035-66ab9afc74f5] start: already running 2023-12-10T19:16:35.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi076.stderr:2023-12-10T19:16:35.143+0000 7f3eaba39640 -1 rbd::mirror::ImageReplayer: 0x55b33ae5a500 [3/c911f848-6f73-4853-a1d8-243e89af66d6] start: already running 2023-12-10T19:16:35.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi076.stderr:2023-12-10T19:16:35.143+0000 7f3eaba39640 -1 rbd::mirror::ImageReplayer: 0x55b33c330500 [3/b1ff34f6-63f9-49d9-9315-4eabc97c393e] start: already running 2023-12-10T19:16:35.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi076.stderr:2023-12-10T19:16:35.143+0000 7f3eaba39640 -1 rbd::mirror::ImageReplayer: 0x55b33ae5ac80 [3/fc4fbc1c-ab50-48d4-a59d-0c60a31269a4] start: already running 2023-12-10T19:16:35.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi076.stderr:2023-12-10T19:16:35.151+0000 7f88ff630640 -1 rbd::mirror::ImageReplayer: 0x55cbc34cb900 [3/4d4eb965-6ed2-4ccb-a76d-8e89ea006260] start: already running 2023-12-10T19:16:35.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi076.stderr:2023-12-10T19:16:35.167+0000 7f3f0da20640 -1 rbd::mirror::ImageReplayer: 0x5620e4936780 [3/0caf4caf-9ee1-436f-874d-71291f8008fb] start: already running 2023-12-10T19:16:35.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi076.stderr:2023-12-10T19:16:35.167+0000 7f3f0da20640 -1 rbd::mirror::ImageReplayer: 0x5620e4937180 [3/1d54e641-de62-4511-9abb-01dd3218783c] start: already running 2023-12-10T19:16:35.327 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi076.stderr:2023-12-10T19:16:35.335+0000 7f3f00205640 -1 rbd::mirror::ImageReplayer: 0x5620e4936c80 [3/195ff7a6-d9ef-4fb6-afac-e87b92563048] start: already running 2023-12-10T19:16:36.056 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_33.status 2023-12-10T19:16:36.057 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:36.057 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:16:36.057 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_33 2023-12-10T19:16:36.057 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_33.status 2023-12-10T19:16:36.057 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2023-12-10T19:16:36.057 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:16:36.122 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:16:36.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=183 2023-12-10T19:16:36.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 183 = 183 2023-12-10T19:16:36.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:16:36.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snap_present cluster1 mirror image_33 snap 2023-12-10T19:16:36.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:36.125 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:16:36.125 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_33 2023-12-10T19:16:36.125 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:16:36.125 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:16:36.125 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:16:37.125 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 info mirror/image_33@snap 2023-12-10T19:16:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_33': 2023-12-10T19:16:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:16:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:16:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 2 2023-12-10T19:16:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 10c0f19c2786 2023-12-10T19:16:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.10c0f19c2786 2023-12-10T19:16:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:16:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering, non-primary 2023-12-10T19:16:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:16:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:16:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:08:23 2023 2023-12-10T19:16:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:08:23 2023 2023-12-10T19:16:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:08:23 2023 2023-12-10T19:16:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: protected: False 2023-12-10T19:16:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:16:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:16:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: 195ff7a6-d9ef-4fb6-afac-e87b92563048 2023-12-10T19:16:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: false 2023-12-10T19:16:37.192 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:16:37.192 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror image_33 snap 2023-12-10T19:16:37.192 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:16:37.192 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_33 2023-12-10T19:16:37.192 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:16:37.192 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:16:37.192 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_33.export 2023-12-10T19:16:37.193 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_33.export 2023-12-10T19:16:37.193 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_33.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_33.export 2023-12-10T19:16:37.193 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export image_33@snap /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_33.export 2023-12-10T19:16:37.417 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:16:37.433 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export image_33@snap /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_33.export 2023-12-10T19:16:37.755 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:16:37.768 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_33.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_33.export 2023-12-10T19:16:37.844 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_33.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_33.export 2023-12-10T19:16:37.883 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:16:37.883 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_34 2023-12-10T19:16:37.883 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror image_34 snap 2023-12-10T19:16:37.883 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:16:37.883 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:16:37.884 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_34 2023-12-10T19:16:37.884 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:16:37.884 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create image_34@snap --debug-rbd=20 --debug-journaler=20 2023-12-10T19:16:38.733 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_started cluster1 mirror image_34 2023-12-10T19:16:38.733 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:38.733 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:16:38.733 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_34 2023-12-10T19:16:38.733 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_state cluster1 mirror image_34 started 2023-12-10T19:16:38.733 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:38.733 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:16:38.733 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_34 2023-12-10T19:16:38.733 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=started 2023-12-10T19:16:38.733 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:16:38.734 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:16:39.734 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_replay_state cluster1 mirror image_34 started 2023-12-10T19:16:39.734 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:39.734 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:16:39.734 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_34 2023-12-10T19:16:39.734 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=started 2023-12-10T19:16:39.734 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_result 2023-12-10T19:16:39.734 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=stopped 2023-12-10T19:16:39.735 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_34 2023-12-10T19:16:39.735 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ + local cluster_instance=cluster1 2023-12-10T19:16:39.735 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:grep -i state 2023-12-10T19:16:39.735 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:16:39.735 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:39.735 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance=cluster1 2023-12-10T19:16:39.735 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loop_instance 2023-12-10T19:16:39.736 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:16:39.736 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T19:16:39.737 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T19:16:39.737 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_34 2023-12-10T19:16:39.737 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T19:16:39.738 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:16:39.738 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:16:39.738 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T19:16:39.738 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:16:39.738 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:16:39.738 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:16:39.738 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:39.738 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:16:39.738 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:16:39.738 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:16:39.738 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:39.738 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:16:39.738 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:16:39.738 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T19:16:39.738 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T19:16:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:16:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T19:16:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:16:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:16:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:16:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:16:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:16:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:16:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:16:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:16:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:16:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T19:16:39.758 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:16:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:16:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:16:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_34 2023-12-10T19:16:39.851 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2023-12-10T19:16:39.852 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_34 2023-12-10T19:16:39.852 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:1 2023-12-10T19:16:39.852 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:16:39.852 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:16:39.852 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-10T19:16:39.852 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:16:39.853 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:16:39.853 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:16:39.853 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:39.853 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:16:39.853 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:16:39.853 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:16:39.853 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:39.853 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:16:39.853 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:16:39.854 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-10T19:16:39.854 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:1 2023-12-10T19:16:39.854 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:39.854 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:16:39.854 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-10T19:16:39.855 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:16:39.855 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:16:39.855 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:16:39.855 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:39.855 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:16:39.855 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:16:39.855 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:16:39.855 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:16:39.855 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:16:39.855 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:16:39.855 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-10T19:16:39.876 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:16:39.876 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:16:39.876 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:16:39.876 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_34 2023-12-10T19:16:39.985 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2023-12-10T19:16:39.985 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:16:39.986 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:16:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_34 2023-12-10T19:16:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:2 2023-12-10T19:16:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:16:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:16:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-10T19:16:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:16:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:16:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:16:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:16:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:16:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:16:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:16:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:16:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-10T19:16:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:2 2023-12-10T19:16:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:16:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-10T19:16:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:16:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:16:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:16:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:16:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:16:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:16:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:16:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:16:39.991 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:16:39.991 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-10T19:16:40.012 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:16:40.012 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:16:40.012 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:16:40.013 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_34 2023-12-10T19:16:40.133 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2023-12-10T19:16:40.134 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:16:40.134 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:16:40.136 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_34 2023-12-10T19:16:40.136 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:3 2023-12-10T19:16:40.136 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:16:40.136 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:16:40.136 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-10T19:16:40.137 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:3 2023-12-10T19:16:40.137 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:16:40.137 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:16:40.137 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:40.137 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:16:40.137 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 3 = cluster1:3 ] 2023-12-10T19:16:40.137 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:16:40.137 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:40.137 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=3 2023-12-10T19:16:40.137 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:16:40.137 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-10T19:16:40.137 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:3 2023-12-10T19:16:40.137 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:40.137 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:16:40.137 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-10T19:16:40.137 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:3 2023-12-10T19:16:40.137 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:16:40.138 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:16:40.138 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:40.138 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:16:40.138 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 3 = cluster1:3 ] 2023-12-10T19:16:40.138 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:16:40.138 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:16:40.138 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=3 2023-12-10T19:16:40.138 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:16:40.138 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-10T19:16:40.160 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T19:16:40.161 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T19:16:40.161 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T19:16:40.161 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_34 2023-12-10T19:16:40.283 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2023-12-10T19:16:40.283 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:16:40.283 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:16:40.284 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:16:40.875 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-12-10T19:16:40.875 INFO:teuthology.orchestra.run:waiting for 600 2023-12-10T19:16:40.876 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-12-10T19:16:40.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-10T19:16:40.876 DEBUG:teuthology.orchestra.run.smithi076:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-10T19:16:40.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-10T19:16:40.879 INFO:tasks.rbd_mirror_thrash:waiting for 113 secs before thrashing 2023-12-10T19:16:41.286 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T19:16:41.286 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T19:16:41.288 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_34 2023-12-10T19:16:41.288 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T19:16:41.288 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:16:41.288 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:16:41.288 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T19:16:41.288 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:16:41.288 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:16:41.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:16:41.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:41.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:16:41.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:16:41.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:16:41.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:41.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:16:41.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:16:41.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T19:16:41.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T19:16:41.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:41.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:16:41.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T19:16:41.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:16:41.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:16:41.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:16:41.290 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:41.290 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:16:41.290 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:16:41.290 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:16:41.290 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:16:41.290 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:16:41.290 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:16:41.290 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T19:16:41.314 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:16:41.314 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:16:41.314 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:16:41.314 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_34 2023-12-10T19:16:41.426 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2023-12-10T19:16:41.426 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:16:41.426 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:16:41.428 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_34 2023-12-10T19:16:41.428 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:1 2023-12-10T19:16:41.428 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:16:41.428 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:16:41.428 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-10T19:16:41.428 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:16:41.428 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:16:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:16:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:16:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:16:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:16:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:16:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:16:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-10T19:16:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:1 2023-12-10T19:16:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:16:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-10T19:16:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:16:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:16:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:16:41.430 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:41.430 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:16:41.430 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:16:41.430 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:16:41.430 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:16:41.430 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:16:41.430 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:16:41.430 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-10T19:16:41.454 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:16:41.454 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:16:41.454 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:16:41.454 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_34 2023-12-10T19:16:41.573 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2023-12-10T19:16:41.573 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:16:41.573 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:16:41.575 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_34 2023-12-10T19:16:41.575 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:2 2023-12-10T19:16:41.575 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:16:41.575 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:16:41.575 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-10T19:16:41.576 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:16:41.576 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:16:41.576 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:16:41.576 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:41.576 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:16:41.576 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:16:41.576 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:16:41.576 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:41.576 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:16:41.576 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:16:41.576 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-10T19:16:41.576 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:2 2023-12-10T19:16:41.576 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:41.576 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:16:41.576 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-10T19:16:41.576 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:16:41.577 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:16:41.577 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:16:41.577 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:41.577 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:16:41.577 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:16:41.577 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:16:41.577 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:16:41.577 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:16:41.577 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:16:41.577 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-10T19:16:41.600 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:16:41.600 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:16:41.600 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:16:41.600 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_34 2023-12-10T19:16:41.721 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2023-12-10T19:16:41.722 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:16:41.722 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:16:41.724 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_34 2023-12-10T19:16:41.724 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:3 2023-12-10T19:16:41.724 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:16:41.724 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:16:41.724 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-10T19:16:41.724 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:3 2023-12-10T19:16:41.724 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:16:41.724 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:16:41.724 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:41.724 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:16:41.724 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 3 = cluster1:3 ] 2023-12-10T19:16:41.724 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:16:41.725 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:41.725 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=3 2023-12-10T19:16:41.725 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:16:41.725 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-10T19:16:41.725 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:3 2023-12-10T19:16:41.725 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:41.725 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:16:41.725 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-10T19:16:41.725 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:3 2023-12-10T19:16:41.725 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:16:41.725 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:16:41.725 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:41.725 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:16:41.725 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 3 = cluster1:3 ] 2023-12-10T19:16:41.725 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:16:41.726 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:16:41.726 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=3 2023-12-10T19:16:41.726 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:16:41.726 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-10T19:16:41.749 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T19:16:41.749 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T19:16:41.749 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T19:16:41.749 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_34 2023-12-10T19:16:41.870 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2023-12-10T19:16:41.871 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:16:41.871 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:16:41.872 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 2 2023-12-10T19:16:43.874 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T19:16:43.874 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T19:16:43.875 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_34 2023-12-10T19:16:43.875 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T19:16:43.875 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:16:43.876 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:16:43.876 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T19:16:43.876 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:16:43.876 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:16:43.876 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:16:43.876 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:43.876 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:16:43.876 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:16:43.876 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:16:43.876 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:43.876 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:16:43.876 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:16:43.876 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T19:16:43.876 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T19:16:43.876 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:43.877 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:16:43.877 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T19:16:43.877 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:16:43.877 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:16:43.877 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:16:43.877 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:43.877 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:16:43.877 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:16:43.877 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:16:43.877 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:16:43.877 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:16:43.877 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:16:43.877 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T19:16:43.901 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:16:43.902 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:16:43.902 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:16:43.902 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_34 2023-12-10T19:16:44.018 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2023-12-10T19:16:44.018 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:16:44.019 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:16:44.021 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_34 2023-12-10T19:16:44.021 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:1 2023-12-10T19:16:44.021 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:16:44.021 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:16:44.021 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-10T19:16:44.021 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:16:44.021 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:16:44.021 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:16:44.021 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:44.021 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:16:44.021 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:16:44.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:16:44.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:44.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:16:44.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:16:44.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-10T19:16:44.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:1 2023-12-10T19:16:44.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:44.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:16:44.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-10T19:16:44.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:16:44.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:16:44.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:16:44.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:44.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:16:44.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:16:44.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:16:44.023 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:16:44.023 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:16:44.023 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:16:44.023 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-10T19:16:44.045 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:16:44.045 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:16:44.045 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:16:44.045 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_34 2023-12-10T19:16:44.153 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2023-12-10T19:16:44.154 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:16:44.154 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:16:44.156 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_34 2023-12-10T19:16:44.156 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:2 2023-12-10T19:16:44.156 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:16:44.156 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:16:44.156 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-10T19:16:44.156 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:16:44.156 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:16:44.156 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:16:44.156 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:44.156 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:16:44.156 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:16:44.156 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:16:44.157 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:44.157 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:16:44.157 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:16:44.157 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-10T19:16:44.157 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:2 2023-12-10T19:16:44.157 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:44.157 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:16:44.157 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-10T19:16:44.157 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:16:44.157 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:16:44.157 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:16:44.157 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:44.157 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:16:44.157 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:16:44.157 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:16:44.157 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:16:44.158 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:16:44.158 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:16:44.158 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-10T19:16:44.180 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:16:44.180 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:16:44.180 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:16:44.181 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_34 2023-12-10T19:16:44.303 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2023-12-10T19:16:44.303 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:16:44.303 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:16:44.306 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_34 2023-12-10T19:16:44.306 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:3 2023-12-10T19:16:44.306 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:16:44.306 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:16:44.306 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-10T19:16:44.306 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:3 2023-12-10T19:16:44.306 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:16:44.307 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:16:44.307 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:44.307 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:16:44.307 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 3 = cluster1:3 ] 2023-12-10T19:16:44.307 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:16:44.307 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:44.307 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=3 2023-12-10T19:16:44.307 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:16:44.307 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-10T19:16:44.307 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:3 2023-12-10T19:16:44.307 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:44.307 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:16:44.307 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-10T19:16:44.307 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:3 2023-12-10T19:16:44.307 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:16:44.308 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:16:44.308 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:44.308 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:16:44.308 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 3 = cluster1:3 ] 2023-12-10T19:16:44.308 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:16:44.308 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:16:44.308 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=3 2023-12-10T19:16:44.308 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:16:44.308 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-10T19:16:44.324 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T19:16:44.325 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T19:16:44.325 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T19:16:44.325 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_34 2023-12-10T19:16:44.440 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2023-12-10T19:16:44.441 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:16:44.441 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:16:44.442 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 4 2023-12-10T19:16:48.444 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T19:16:48.444 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T19:16:48.445 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_34 2023-12-10T19:16:48.446 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T19:16:48.446 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:16:48.446 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:16:48.446 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T19:16:48.446 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:16:48.446 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:16:48.446 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:16:48.446 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:48.446 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:16:48.446 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:16:48.446 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:16:48.446 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:48.446 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:16:48.447 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:16:48.447 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T19:16:48.447 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T19:16:48.447 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:48.447 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:16:48.447 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T19:16:48.447 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:16:48.447 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:16:48.447 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:16:48.447 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:48.447 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:16:48.447 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:16:48.447 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:16:48.447 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:16:48.447 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:16:48.448 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:16:48.448 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T19:16:48.472 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:16:48.473 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:16:48.473 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:16:48.473 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_34 2023-12-10T19:16:48.588 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2023-12-10T19:16:48.589 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:16:48.589 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:16:48.591 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_34 2023-12-10T19:16:48.591 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:1 2023-12-10T19:16:48.591 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:16:48.591 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:16:48.591 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-10T19:16:48.591 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:16:48.591 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:16:48.591 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:16:48.591 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:48.591 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:16:48.592 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:16:48.592 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:16:48.592 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:48.592 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:16:48.592 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:16:48.592 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-10T19:16:48.592 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:1 2023-12-10T19:16:48.592 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:48.592 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:16:48.592 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-10T19:16:48.592 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:16:48.592 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:16:48.592 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:16:48.592 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:48.592 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:16:48.593 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:16:48.593 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:16:48.593 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:16:48.593 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:16:48.594 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:16:48.594 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-10T19:16:48.616 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:16:48.616 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:16:48.616 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:16:48.616 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_34 2023-12-10T19:16:48.724 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2023-12-10T19:16:48.724 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:16:48.724 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:16:48.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_34 2023-12-10T19:16:48.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:2 2023-12-10T19:16:48.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:16:48.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:16:48.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-10T19:16:48.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:16:48.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:16:48.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:16:48.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:48.728 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:16:48.728 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:16:48.728 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:16:48.728 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:48.728 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:16:48.728 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:16:48.728 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-10T19:16:48.728 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:2 2023-12-10T19:16:48.728 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:48.728 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:16:48.728 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-10T19:16:48.728 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:16:48.728 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:16:48.728 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:16:48.728 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:48.729 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:16:48.729 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:16:48.729 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:16:48.729 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:16:48.729 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:16:48.729 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:16:48.729 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-10T19:16:48.746 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:16:48.746 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:16:48.746 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:16:48.746 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_34 2023-12-10T19:16:48.852 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2023-12-10T19:16:48.852 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:16:48.852 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:16:48.855 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_34 2023-12-10T19:16:48.855 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:3 2023-12-10T19:16:48.855 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:16:48.855 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:16:48.855 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-10T19:16:48.855 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:3 2023-12-10T19:16:48.856 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:16:48.856 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:16:48.856 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:48.856 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:16:48.856 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 3 = cluster1:3 ] 2023-12-10T19:16:48.856 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:16:48.856 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:48.856 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=3 2023-12-10T19:16:48.856 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:16:48.856 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-10T19:16:48.856 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:3 2023-12-10T19:16:48.856 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:48.856 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:16:48.856 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-10T19:16:48.856 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:3 2023-12-10T19:16:48.857 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:16:48.857 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:16:48.857 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:48.857 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:16:48.857 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 3 = cluster1:3 ] 2023-12-10T19:16:48.857 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:16:48.857 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:16:48.857 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=3 2023-12-10T19:16:48.857 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:16:48.857 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-10T19:16:48.879 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T19:16:48.879 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T19:16:48.879 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T19:16:48.879 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_34 2023-12-10T19:16:48.984 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2023-12-10T19:16:48.984 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:16:48.984 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:16:48.985 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 8 2023-12-10T19:16:56.987 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T19:16:56.987 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T19:16:56.989 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_34 2023-12-10T19:16:56.989 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T19:16:56.989 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:16:56.989 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:16:56.989 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T19:16:56.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:16:56.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:16:56.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:16:56.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:56.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:16:56.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:16:56.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:16:56.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:56.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:16:56.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:16:56.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T19:16:56.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T19:16:56.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:56.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:16:56.991 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T19:16:56.991 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:16:56.991 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:16:56.991 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:16:56.991 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:56.991 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:16:56.991 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:16:56.991 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:16:56.991 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:16:56.991 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:16:56.991 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:16:56.991 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T19:16:57.015 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:16:57.015 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:16:57.015 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:16:57.016 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_34 2023-12-10T19:16:57.127 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2023-12-10T19:16:57.127 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:16:57.127 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:16:57.130 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_34 2023-12-10T19:16:57.130 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:1 2023-12-10T19:16:57.130 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:16:57.130 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:16:57.130 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-10T19:16:57.130 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:16:57.130 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:16:57.130 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:16:57.130 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:57.130 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:16:57.130 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:16:57.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:16:57.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:57.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:16:57.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:16:57.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-10T19:16:57.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:1 2023-12-10T19:16:57.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:57.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:16:57.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-10T19:16:57.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:16:57.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:16:57.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:16:57.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:57.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:16:57.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:16:57.132 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:16:57.132 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:16:57.132 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:16:57.132 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:16:57.132 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-10T19:16:57.154 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:16:57.154 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:16:57.154 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:16:57.154 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_34 2023-12-10T19:16:57.269 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2023-12-10T19:16:57.269 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:16:57.269 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:16:57.271 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_34 2023-12-10T19:16:57.271 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:2 2023-12-10T19:16:57.271 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:16:57.271 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:16:57.271 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-10T19:16:57.271 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:16:57.271 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:16:57.272 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:16:57.272 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:57.272 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:16:57.272 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:16:57.272 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:16:57.272 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:57.272 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:16:57.272 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:16:57.272 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-10T19:16:57.272 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:2 2023-12-10T19:16:57.272 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:57.272 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:16:57.272 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-10T19:16:57.272 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:16:57.272 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:16:57.272 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:16:57.273 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:57.273 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:16:57.273 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:16:57.273 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:16:57.273 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:16:57.273 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:16:57.273 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:16:57.273 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-10T19:16:57.296 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:16:57.297 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:16:57.297 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:16:57.297 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_34 2023-12-10T19:16:57.412 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2023-12-10T19:16:57.412 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:16:57.412 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:16:57.415 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_34 2023-12-10T19:16:57.415 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:3 2023-12-10T19:16:57.415 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:16:57.415 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:16:57.415 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-10T19:16:57.415 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:3 2023-12-10T19:16:57.415 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:16:57.415 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:16:57.415 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:57.416 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:16:57.416 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 3 = cluster1:3 ] 2023-12-10T19:16:57.416 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:16:57.416 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:57.416 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=3 2023-12-10T19:16:57.416 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:16:57.416 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-10T19:16:57.416 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:3 2023-12-10T19:16:57.416 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:16:57.416 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:16:57.416 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-10T19:16:57.416 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:3 2023-12-10T19:16:57.416 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:16:57.417 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:16:57.417 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:16:57.417 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:16:57.417 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 3 = cluster1:3 ] 2023-12-10T19:16:57.417 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:16:57.417 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:16:57.417 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=3 2023-12-10T19:16:57.417 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:16:57.417 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-10T19:16:57.438 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T19:16:57.438 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T19:16:57.438 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T19:16:57.438 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_34 2023-12-10T19:16:57.556 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2023-12-10T19:16:57.556 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:16:57.556 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:16:57.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 8 2023-12-10T19:17:05.559 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T19:17:05.560 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T19:17:05.561 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_34 2023-12-10T19:17:05.561 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T19:17:05.561 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:17:05.561 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:17:05.561 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T19:17:05.562 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:17:05.562 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:17:05.562 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:17:05.562 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:17:05.562 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:17:05.562 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:17:05.562 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:17:05.562 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:17:05.562 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:17:05.562 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:17:05.562 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T19:17:05.562 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T19:17:05.562 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:05.562 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:17:05.562 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T19:17:05.563 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:17:05.563 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:17:05.563 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:17:05.563 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:17:05.563 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:17:05.563 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:17:05.563 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:17:05.563 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:17:05.563 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:17:05.563 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:17:05.563 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T19:17:05.587 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:17:05.587 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:17:05.587 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:17:05.588 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_34 2023-12-10T19:17:05.699 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2023-12-10T19:17:05.699 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:17:05.699 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:17:05.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_34 2023-12-10T19:17:05.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:1 2023-12-10T19:17:05.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:17:05.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:17:05.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-10T19:17:05.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:17:05.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:17:05.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:17:05.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:17:05.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:17:05.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:17:05.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:17:05.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:17:05.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:17:05.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:17:05.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-10T19:17:05.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:1 2023-12-10T19:17:05.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:05.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:17:05.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-10T19:17:05.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:17:05.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:17:05.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:17:05.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:17:05.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:17:05.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:17:05.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:17:05.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:17:05.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:17:05.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:17:05.704 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-10T19:17:05.725 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:17:05.725 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:17:05.725 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:17:05.725 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_34 2023-12-10T19:17:05.843 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2023-12-10T19:17:05.843 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:17:05.843 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:17:05.845 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_34 2023-12-10T19:17:05.846 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:2 2023-12-10T19:17:05.846 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:17:05.846 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:17:05.846 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-10T19:17:05.846 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:17:05.846 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:17:05.846 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:17:05.846 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:17:05.846 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:17:05.846 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:17:05.846 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:17:05.846 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:17:05.846 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:17:05.847 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:17:05.847 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-10T19:17:05.847 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:2 2023-12-10T19:17:05.847 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:05.847 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:17:05.847 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-10T19:17:05.847 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:17:05.847 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:17:05.847 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:17:05.847 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:17:05.847 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:17:05.847 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:17:05.847 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:17:05.847 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:17:05.847 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:17:05.847 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:17:05.848 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-10T19:17:05.871 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:17:05.871 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:17:05.871 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:17:05.871 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_34 2023-12-10T19:17:05.984 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2023-12-10T19:17:05.984 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:17:05.984 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:17:05.989 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_34 2023-12-10T19:17:05.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:3 2023-12-10T19:17:05.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:17:05.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:17:05.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-10T19:17:05.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:3 2023-12-10T19:17:05.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:17:05.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:17:05.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:17:05.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:17:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 3 = cluster1:3 ] 2023-12-10T19:17:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:17:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:17:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=3 2023-12-10T19:17:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:17:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-10T19:17:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:3 2023-12-10T19:17:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:17:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-10T19:17:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:3 2023-12-10T19:17:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:17:05.992 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:17:05.992 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:17:05.992 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:17:05.992 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 3 = cluster1:3 ] 2023-12-10T19:17:05.992 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:17:05.992 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:17:05.992 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=3 2023-12-10T19:17:05.992 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:17:05.992 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-10T19:17:06.010 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T19:17:06.010 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T19:17:06.010 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T19:17:06.010 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_34 2023-12-10T19:17:06.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2023-12-10T19:17:06.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:17:06.125 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:17:06.126 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 8 2023-12-10T19:17:14.127 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T19:17:14.128 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T19:17:14.130 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_34 2023-12-10T19:17:14.130 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T19:17:14.130 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:17:14.130 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:17:14.130 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T19:17:14.130 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:17:14.130 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:17:14.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:17:14.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:17:14.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:17:14.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:17:14.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:17:14.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:17:14.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:17:14.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:17:14.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T19:17:14.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T19:17:14.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:14.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:17:14.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T19:17:14.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:17:14.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:17:14.132 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:17:14.132 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:17:14.132 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:17:14.132 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:17:14.132 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:17:14.132 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:17:14.132 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:17:14.132 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:17:14.132 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T19:17:14.156 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:17:14.156 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:17:14.156 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:17:14.156 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_34 2023-12-10T19:17:14.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:17:14.290 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ status_result= "state": "Replaying" 2023-12-10T19:17:14.290 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo "state": "Replaying" 2023-12-10T19:17:14.290 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep -i Replaying 2023-12-10T19:17:14.291 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "state": "Replaying" 2023-12-10T19:17:14.293 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=started 2023-12-10T19:17:14.294 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test started = started 2023-12-10T19:17:14.294 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:17:14.294 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_34 2023-12-10T19:17:14.294 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:17:14.294 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:17:14.294 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:14.294 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_34 2023-12-10T19:17:14.294 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = journal ] 2023-12-10T19:17:14.294 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = snapshot ] 2023-12-10T19:17:14.294 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_34 2023-12-10T19:17:14.294 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:17:14.294 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:17:14.294 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:14.294 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_34 2023-12-10T19:17:14.295 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster2-mirror-image_34.status 2023-12-10T19:17:14.295 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ + echo cluster2-mirror-image_34.status 2023-12-10T19:17:14.295 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:sed -e s|[/ ]|_|g 2023-12-10T19:17:14.295 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_34.status 2023-12-10T19:17:14.295 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-image_34.status 2023-12-10T19:17:14.295 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-image_34.status 2023-12-10T19:17:14.295 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:17:14.296 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_34.status 2023-12-10T19:17:14.296 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mirror_image_snapshot cluster2 mirror image_34 2023-12-10T19:17:14.296 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:17:14.296 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:14.296 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_34 2023-12-10T19:17:14.296 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_34 2023-12-10T19:17:14.766 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Snapshot ID: 185 2023-12-10T19:17:14.780 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_34 /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_34.status 2023-12-10T19:17:14.781 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:17:14.781 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:14.781 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_34 2023-12-10T19:17:14.781 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_34.status 2023-12-10T19:17:14.781 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_34 --format xml 2023-12-10T19:17:14.781 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:17:14.837 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-10T19:17:14.839 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snapshot_id=185 2023-12-10T19:17:14.839 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:17:14.840 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.2 2023-12-10T19:17:15.041 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_34.status 2023-12-10T19:17:15.041 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:15.041 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:15.042 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_34 2023-12-10T19:17:15.042 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_34.status 2023-12-10T19:17:15.042 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2023-12-10T19:17:15.042 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:17:15.108 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:17:15.109 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:17:15.110 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:17:15.110 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:17:15.110 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:17:15.111 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:17:15.111 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 185 = 2023-12-10T19:17:15.111 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.4 2023-12-10T19:17:15.512 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_34.status 2023-12-10T19:17:15.513 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:15.513 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:15.513 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_34 2023-12-10T19:17:15.513 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_34.status 2023-12-10T19:17:15.513 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2023-12-10T19:17:15.513 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:17:15.582 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:17:15.582 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:17:15.584 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:17:15.584 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:17:15.584 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:17:15.584 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:17:15.584 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 185 = 2023-12-10T19:17:15.584 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.8 2023-12-10T19:17:16.385 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_34.status 2023-12-10T19:17:16.385 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:16.385 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:16.385 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_34 2023-12-10T19:17:16.386 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_34.status 2023-12-10T19:17:16.386 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2023-12-10T19:17:16.386 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:17:16.456 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:17:16.458 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=185 2023-12-10T19:17:16.459 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 185 = 185 2023-12-10T19:17:16.459 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:17:16.459 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snap_present cluster1 mirror image_34 snap 2023-12-10T19:17:16.459 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:16.459 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:16.459 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_34 2023-12-10T19:17:16.459 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:17:16.459 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:17:16.459 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:17:17.459 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 info mirror/image_34@snap 2023-12-10T19:17:17.520 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_34': 2023-12-10T19:17:17.520 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:17:17.520 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:17:17.520 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 2 2023-12-10T19:17:17.520 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 12a87b8efdfb 2023-12-10T19:17:17.520 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.12a87b8efdfb 2023-12-10T19:17:17.520 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:17:17.520 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering, non-primary 2023-12-10T19:17:17.520 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:17:17.520 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:17:17.520 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:08:35 2023 2023-12-10T19:17:17.521 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:08:35 2023 2023-12-10T19:17:17.521 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:08:35 2023 2023-12-10T19:17:17.521 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: protected: False 2023-12-10T19:17:17.521 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:17:17.521 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:17:17.521 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: d25b8883-1661-4a4c-bc7a-8d186c1f1d26 2023-12-10T19:17:17.521 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: false 2023-12-10T19:17:17.525 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:17:17.525 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror image_34 snap 2023-12-10T19:17:17.525 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:17.526 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_34 2023-12-10T19:17:17.526 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:17:17.526 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:17:17.526 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_34.export 2023-12-10T19:17:17.526 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_34.export 2023-12-10T19:17:17.526 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_34.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_34.export 2023-12-10T19:17:17.527 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export image_34@snap /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_34.export 2023-12-10T19:17:17.735 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:17:17.742 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export image_34@snap /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_34.export 2023-12-10T19:17:18.084 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:17:18.099 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_34.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_34.export 2023-12-10T19:17:18.176 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_34.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_34.export 2023-12-10T19:17:18.216 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:17:18.217 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_35 2023-12-10T19:17:18.217 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror image_35 snap 2023-12-10T19:17:18.217 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:17:18.217 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:18.217 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_35 2023-12-10T19:17:18.217 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:17:18.217 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create image_35@snap --debug-rbd=20 --debug-journaler=20 2023-12-10T19:17:18.758 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_started cluster1 mirror image_35 2023-12-10T19:17:18.758 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:18.758 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:18.758 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_35 2023-12-10T19:17:18.758 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_state cluster1 mirror image_35 started 2023-12-10T19:17:18.758 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:18.758 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:18.758 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_35 2023-12-10T19:17:18.759 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=started 2023-12-10T19:17:18.759 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:17:18.759 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:17:19.759 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_replay_state cluster1 mirror image_35 started 2023-12-10T19:17:19.759 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:19.759 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:19.759 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_35 2023-12-10T19:17:19.759 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=started 2023-12-10T19:17:19.759 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_result 2023-12-10T19:17:19.759 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=stopped 2023-12-10T19:17:19.760 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemons+ cluster1 rbd mirror status mirror/image_35 2023-12-10T19:17:19.760 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:grep -i state 2023-12-10T19:17:19.760 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_instance=cluster1 2023-12-10T19:17:19.760 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:17:19.760 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:19.760 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance=cluster1 2023-12-10T19:17:19.760 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loop_instance 2023-12-10T19:17:19.760 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:17:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T19:17:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T19:17:19.763 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_35 2023-12-10T19:17:19.763 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T19:17:19.763 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:17:19.763 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:17:19.763 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T19:17:19.763 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:17:19.763 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:17:19.763 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:17:19.763 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:17:19.763 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:17:19.763 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:17:19.763 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:17:19.763 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:17:19.764 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:17:19.764 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:17:19.764 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T19:17:19.764 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T19:17:19.764 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:19.764 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:17:19.764 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T19:17:19.764 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:17:19.764 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:17:19.764 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:17:19.764 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:17:19.764 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:17:19.764 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:17:19.764 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:17:19.764 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:17:19.764 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:17:19.765 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:17:19.765 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T19:17:19.789 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:17:19.789 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:17:19.789 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:17:19.789 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_35 2023-12-10T19:17:19.902 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_35'] 2023-12-10T19:17:19.902 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:17:19.902 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:17:19.905 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_35 2023-12-10T19:17:19.905 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:1 2023-12-10T19:17:19.905 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:17:19.905 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:17:19.905 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-10T19:17:19.905 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:17:19.905 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:17:19.905 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:17:19.905 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:17:19.905 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:17:19.905 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:17:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:17:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:17:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:17:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:17:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-10T19:17:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:1 2023-12-10T19:17:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:17:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-10T19:17:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:17:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:17:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:17:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:17:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:17:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:17:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:17:19.907 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:17:19.907 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:17:19.907 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:17:19.907 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-10T19:17:19.928 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:17:19.929 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:17:19.929 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:17:19.929 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_35 2023-12-10T19:17:20.051 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:17:20.052 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ status_result= "state": "Replaying" 2023-12-10T19:17:20.052 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ + echo "state": "Replaying" 2023-12-10T19:17:20.052 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:grep -i Replaying 2023-12-10T19:17:20.053 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "state": "Replaying" 2023-12-10T19:17:20.054 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=started 2023-12-10T19:17:20.054 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test started = started 2023-12-10T19:17:20.055 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:17:20.055 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_35 2023-12-10T19:17:20.055 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:17:20.055 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:17:20.055 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:20.055 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_35 2023-12-10T19:17:20.055 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = journal ] 2023-12-10T19:17:20.055 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = snapshot ] 2023-12-10T19:17:20.055 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_35 2023-12-10T19:17:20.055 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:17:20.055 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:17:20.055 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:20.055 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_35 2023-12-10T19:17:20.055 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster2-mirror-image_35.status 2023-12-10T19:17:20.055 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2-mirror-image_35.status 2023-12-10T19:17:20.055 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:17:20.056 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_35.status 2023-12-10T19:17:20.056 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-image_35.status 2023-12-10T19:17:20.057 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-image_35.status 2023-12-10T19:17:20.057 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:17:20.057 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_35.status 2023-12-10T19:17:20.057 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mirror_image_snapshot cluster2 mirror image_35 2023-12-10T19:17:20.057 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:17:20.057 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:20.057 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_35 2023-12-10T19:17:20.057 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_35 2023-12-10T19:17:20.795 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Snapshot ID: 187 2023-12-10T19:17:20.806 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_35 /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_35.status 2023-12-10T19:17:20.806 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:17:20.806 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:20.806 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_35 2023-12-10T19:17:20.806 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_35.status 2023-12-10T19:17:20.806 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_35 --format xml 2023-12-10T19:17:20.807 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:17:20.857 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-10T19:17:20.860 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snapshot_id=187 2023-12-10T19:17:20.860 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:17:20.860 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.2 2023-12-10T19:17:21.062 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_35.status 2023-12-10T19:17:21.062 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:21.062 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:21.062 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_35 2023-12-10T19:17:21.062 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_35.status 2023-12-10T19:17:21.062 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2023-12-10T19:17:21.062 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:17:21.127 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:17:21.127 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:17:21.130 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:17:21.130 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:17:21.130 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:17:21.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:17:21.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 187 = 2023-12-10T19:17:21.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.4 2023-12-10T19:17:21.532 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_35.status 2023-12-10T19:17:21.532 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:21.532 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:21.532 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_35 2023-12-10T19:17:21.532 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_35.status 2023-12-10T19:17:21.532 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2023-12-10T19:17:21.533 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:17:21.600 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:17:21.600 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:17:21.603 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:17:21.604 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:17:21.604 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:17:21.604 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:17:21.604 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 187 = 2023-12-10T19:17:21.604 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.8 2023-12-10T19:17:22.405 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_35.status 2023-12-10T19:17:22.406 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:22.406 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:22.406 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_35 2023-12-10T19:17:22.406 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_35.status 2023-12-10T19:17:22.406 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2023-12-10T19:17:22.406 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:17:22.476 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:17:22.480 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=187 2023-12-10T19:17:22.480 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 187 = 187 2023-12-10T19:17:22.480 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:17:22.480 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snap_present cluster1 mirror image_35 snap 2023-12-10T19:17:22.480 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:22.480 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:22.480 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_35 2023-12-10T19:17:22.480 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:17:22.480 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:17:22.480 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:17:23.480 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 info mirror/image_35@snap 2023-12-10T19:17:23.542 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_35': 2023-12-10T19:17:23.542 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:17:23.542 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:17:23.543 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 2 2023-12-10T19:17:23.543 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 12a8a92d8260 2023-12-10T19:17:23.543 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.12a8a92d8260 2023-12-10T19:17:23.543 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:17:23.543 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering, non-primary 2023-12-10T19:17:23.543 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:17:23.543 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:17:23.543 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:08:47 2023 2023-12-10T19:17:23.543 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:08:47 2023 2023-12-10T19:17:23.543 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:08:47 2023 2023-12-10T19:17:23.543 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: protected: False 2023-12-10T19:17:23.543 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:17:23.544 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:17:23.544 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: fc4fbc1c-ab50-48d4-a59d-0c60a31269a4 2023-12-10T19:17:23.544 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: false 2023-12-10T19:17:23.547 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:17:23.548 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror image_35 snap 2023-12-10T19:17:23.548 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:23.548 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_35 2023-12-10T19:17:23.548 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:17:23.548 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:17:23.548 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_35.export 2023-12-10T19:17:23.549 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_35.export 2023-12-10T19:17:23.549 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_35.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_35.export 2023-12-10T19:17:23.549 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export image_35@snap /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_35.export 2023-12-10T19:17:23.741 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:17:23.748 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export image_35@snap /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_35.export 2023-12-10T19:17:24.001 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:17:24.016 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_35.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_35.export 2023-12-10T19:17:24.092 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_35.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_35.export 2023-12-10T19:17:24.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:17:24.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_36 2023-12-10T19:17:24.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror image_36 snap 2023-12-10T19:17:24.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:17:24.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:24.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_36 2023-12-10T19:17:24.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:17:24.131 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create image_36@snap --debug-rbd=20 --debug-journaler=20 2023-12-10T19:17:24.781 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_started cluster1 mirror image_36 2023-12-10T19:17:24.781 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:24.781 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:24.781 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_36 2023-12-10T19:17:24.781 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_state cluster1 mirror image_36 started 2023-12-10T19:17:24.782 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:24.782 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:24.782 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_36 2023-12-10T19:17:24.782 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=started 2023-12-10T19:17:24.782 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:17:24.782 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:17:25.782 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_replay_state cluster1 mirror image_36 started 2023-12-10T19:17:25.782 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:25.782 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:25.782 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_36 2023-12-10T19:17:25.783 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=started 2023-12-10T19:17:25.783 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_result 2023-12-10T19:17:25.783 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=stopped 2023-12-10T19:17:25.784 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_36 2023-12-10T19:17:25.784 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ + local cluster_instance=cluster1 2023-12-10T19:17:25.784 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:grep -i state+ 2023-12-10T19:17:25.784 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:shift 2023-12-10T19:17:25.784 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:25.784 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance=cluster1 2023-12-10T19:17:25.784 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loop_instance 2023-12-10T19:17:25.784 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:17:25.785 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T19:17:25.786 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T19:17:25.786 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_36 2023-12-10T19:17:25.786 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T19:17:25.786 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:17:25.786 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:17:25.786 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T19:17:25.786 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:17:25.786 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:17:25.786 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:17:25.787 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:17:25.787 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:17:25.787 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:17:25.787 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:17:25.787 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:17:25.787 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:17:25.787 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:17:25.787 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T19:17:25.787 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T19:17:25.787 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:25.787 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:17:25.787 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T19:17:25.787 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:17:25.787 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:17:25.787 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:17:25.787 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:17:25.788 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:17:25.788 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:17:25.788 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:17:25.788 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:17:25.788 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:17:25.788 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:17:25.788 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T19:17:25.807 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:17:25.807 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:17:25.807 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:17:25.807 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_36 2023-12-10T19:17:25.919 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2023-12-10T19:17:25.919 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:17:25.919 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:17:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_36 2023-12-10T19:17:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:1 2023-12-10T19:17:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:17:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:17:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-10T19:17:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:17:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:17:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:17:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:17:25.922 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:17:25.922 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:17:25.922 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:17:25.922 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:17:25.922 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:17:25.922 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:17:25.922 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-10T19:17:25.922 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:1 2023-12-10T19:17:25.922 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:25.922 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:17:25.922 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-10T19:17:25.922 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:17:25.922 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:17:25.922 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:17:25.922 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:17:25.923 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:17:25.923 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:17:25.923 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:17:25.923 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:17:25.923 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:17:25.923 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:17:25.923 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-10T19:17:25.944 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:17:25.945 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:17:25.945 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:17:25.945 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_36 2023-12-10T19:17:26.054 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2023-12-10T19:17:26.055 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:17:26.055 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:17:26.058 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_36 2023-12-10T19:17:26.058 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:2 2023-12-10T19:17:26.058 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:17:26.058 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:17:26.058 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-10T19:17:26.058 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:17:26.058 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:17:26.058 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:17:26.058 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:17:26.058 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:17:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:17:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:17:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:17:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:17:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:17:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-10T19:17:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:2 2023-12-10T19:17:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:17:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-10T19:17:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:17:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:17:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:17:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:17:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:17:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:17:26.060 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:17:26.060 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:17:26.060 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:17:26.060 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:17:26.060 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-10T19:17:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:17:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:17:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:17:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_36 2023-12-10T19:17:26.199 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:17:26.199 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ status_result= "state": "Replaying" 2023-12-10T19:17:26.199 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo "state": "Replaying" 2023-12-10T19:17:26.199 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep -i Replaying 2023-12-10T19:17:26.201 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "state": "Replaying" 2023-12-10T19:17:26.203 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=started 2023-12-10T19:17:26.203 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test started = started 2023-12-10T19:17:26.203 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:17:26.203 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_36 2023-12-10T19:17:26.203 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:17:26.203 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:17:26.203 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:26.203 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_36 2023-12-10T19:17:26.203 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = journal ] 2023-12-10T19:17:26.203 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = snapshot ] 2023-12-10T19:17:26.204 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_36 2023-12-10T19:17:26.204 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:17:26.204 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:17:26.204 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:26.204 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_36 2023-12-10T19:17:26.204 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster2-mirror-image_36.status 2023-12-10T19:17:26.204 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2-mirror-image_36.status 2023-12-10T19:17:26.204 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:17:26.204 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_36.status 2023-12-10T19:17:26.204 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-image_36.status 2023-12-10T19:17:26.204 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-image_36.status 2023-12-10T19:17:26.204 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:17:26.205 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_36.status 2023-12-10T19:17:26.205 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mirror_image_snapshot cluster2 mirror image_36 2023-12-10T19:17:26.205 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:17:26.205 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:26.205 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_36 2023-12-10T19:17:26.205 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_36 2023-12-10T19:17:26.814 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Snapshot ID: 189 2023-12-10T19:17:26.825 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_36 /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_36.status 2023-12-10T19:17:26.825 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:17:26.825 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:26.825 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_36 2023-12-10T19:17:26.825 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_36.status 2023-12-10T19:17:26.825 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_36 --format xml 2023-12-10T19:17:26.825 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:17:26.877 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-10T19:17:26.879 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snapshot_id=189 2023-12-10T19:17:26.880 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:17:26.880 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.2 2023-12-10T19:17:27.081 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_36.status 2023-12-10T19:17:27.081 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:27.081 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:27.081 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_36 2023-12-10T19:17:27.081 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_36.status 2023-12-10T19:17:27.082 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2023-12-10T19:17:27.082 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:17:27.150 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:17:27.150 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:17:27.152 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:17:27.152 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:17:27.152 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:17:27.152 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:17:27.153 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 189 = 2023-12-10T19:17:27.153 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.4 2023-12-10T19:17:27.553 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_36.status 2023-12-10T19:17:27.554 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:27.554 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:27.554 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_36 2023-12-10T19:17:27.554 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_36.status 2023-12-10T19:17:27.554 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2023-12-10T19:17:27.554 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:17:27.692 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:17:27.693 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:17:27.695 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:17:27.695 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:17:27.696 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:17:27.696 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:17:27.696 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 189 = 2023-12-10T19:17:27.696 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.8 2023-12-10T19:17:27.866 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.985327151446249) > weight (0.25) 2023-12-10T19:17:27.867 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2023-12-10T19:17:27.867 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2023-12-10T19:17:27.867 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2023-12-10T19:17:27.868 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr:2023-12-10T19:17:27.874+0000 7f1926bd5640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 54852) UID: 1000 2023-12-10T19:17:28.498 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_36.status 2023-12-10T19:17:28.498 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:28.498 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:28.498 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_36 2023-12-10T19:17:28.498 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_36.status 2023-12-10T19:17:28.498 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2023-12-10T19:17:28.498 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:17:28.568 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:17:28.571 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=189 2023-12-10T19:17:28.571 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 189 = 189 2023-12-10T19:17:28.571 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:17:28.571 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snap_present cluster1 mirror image_36 snap 2023-12-10T19:17:28.571 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:28.572 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:28.572 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_36 2023-12-10T19:17:28.572 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:17:28.572 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:17:28.572 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:17:29.572 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 info mirror/image_36@snap 2023-12-10T19:17:29.632 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_36': 2023-12-10T19:17:29.632 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:17:29.632 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:17:29.632 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 2 2023-12-10T19:17:29.632 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 12a8f002056f 2023-12-10T19:17:29.632 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.12a8f002056f 2023-12-10T19:17:29.633 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:17:29.633 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering, non-primary 2023-12-10T19:17:29.633 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:17:29.633 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:17:29.633 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:08:54 2023 2023-12-10T19:17:29.633 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:08:54 2023 2023-12-10T19:17:29.633 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:08:54 2023 2023-12-10T19:17:29.633 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: protected: False 2023-12-10T19:17:29.633 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:17:29.633 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:17:29.633 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: ad2cd5bc-da17-44b5-9afd-c05bcf6f8746 2023-12-10T19:17:29.633 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: false 2023-12-10T19:17:29.636 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:17:29.637 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror image_36 snap 2023-12-10T19:17:29.637 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:29.637 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_36 2023-12-10T19:17:29.637 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:17:29.637 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:17:29.637 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_36.export 2023-12-10T19:17:29.637 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_36.export 2023-12-10T19:17:29.637 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_36.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_36.export 2023-12-10T19:17:29.637 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export image_36@snap /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_36.export 2023-12-10T19:17:29.873 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:17:29.884 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export image_36@snap /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_36.export 2023-12-10T19:17:30.222 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:17:30.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_36.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_36.export 2023-12-10T19:17:30.313 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_36.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_36.export 2023-12-10T19:17:30.352 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:17:30.353 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_37 2023-12-10T19:17:30.353 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror image_37 snap 2023-12-10T19:17:30.353 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:17:30.353 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:30.353 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_37 2023-12-10T19:17:30.353 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:17:30.353 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create image_37@snap --debug-rbd=20 --debug-journaler=20 2023-12-10T19:17:30.801 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_started cluster1 mirror image_37 2023-12-10T19:17:30.801 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:30.801 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:30.801 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_37 2023-12-10T19:17:30.801 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_state cluster1 mirror image_37 started 2023-12-10T19:17:30.801 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:30.801 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:30.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_37 2023-12-10T19:17:30.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=started 2023-12-10T19:17:30.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:17:30.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:17:31.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_replay_state cluster1 mirror image_37 started 2023-12-10T19:17:31.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:31.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:31.803 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_37 2023-12-10T19:17:31.803 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=started 2023-12-10T19:17:31.803 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_result 2023-12-10T19:17:31.803 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=stopped 2023-12-10T19:17:31.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_37 2023-12-10T19:17:31.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_instance=cluster1 2023-12-10T19:17:31.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:17:31.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ + local cluster=cluster1 2023-12-10T19:17:31.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance=cluster1 2023-12-10T19:17:31.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:grep+ -ilocal state loop_instance 2023-12-10T19:17:31.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:17:31.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:17:31.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T19:17:31.806 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T19:17:31.806 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_37 2023-12-10T19:17:31.806 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T19:17:31.806 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:17:31.806 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:17:31.806 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T19:17:31.806 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:17:31.806 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:17:31.806 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:17:31.807 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:17:31.807 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:17:31.807 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:17:31.807 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:17:31.807 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:17:31.807 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:17:31.807 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:17:31.807 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T19:17:31.807 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T19:17:31.807 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:31.807 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:17:31.807 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T19:17:31.807 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:17:31.807 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:17:31.807 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:17:31.807 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:17:31.808 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:17:31.808 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:17:31.808 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:17:31.808 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:17:31.808 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:17:31.808 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:17:31.808 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T19:17:31.829 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:17:31.829 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:17:31.829 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:17:31.829 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_37 2023-12-10T19:17:31.941 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_37'] 2023-12-10T19:17:31.941 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:17:31.941 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:17:31.944 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_37 2023-12-10T19:17:31.944 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:1 2023-12-10T19:17:31.944 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:17:31.944 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:17:31.944 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-10T19:17:31.945 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:17:31.945 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:17:31.945 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:17:31.945 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:17:31.945 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:17:31.945 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:17:31.945 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:17:31.945 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:17:31.945 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:17:31.945 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:17:31.945 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-10T19:17:31.945 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:1 2023-12-10T19:17:31.945 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:31.945 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:17:31.945 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-10T19:17:31.946 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:17:31.946 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:17:31.946 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:17:31.946 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:17:31.946 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:17:31.946 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:17:31.946 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:17:31.946 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:17:31.946 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:17:31.946 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:17:31.946 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-10T19:17:31.967 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:17:31.967 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:17:31.967 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:17:31.967 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_37 2023-12-10T19:17:32.103 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:17:32.104 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ status_result= "state": "Replaying" 2023-12-10T19:17:32.104 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo "state": "Replaying" 2023-12-10T19:17:32.104 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep -i Replaying 2023-12-10T19:17:32.105 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "state": "Replaying" 2023-12-10T19:17:32.106 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=started 2023-12-10T19:17:32.106 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test started = started 2023-12-10T19:17:32.107 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:17:32.107 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_37 2023-12-10T19:17:32.107 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:17:32.107 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:17:32.107 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:32.107 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_37 2023-12-10T19:17:32.107 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = journal ] 2023-12-10T19:17:32.107 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = snapshot ] 2023-12-10T19:17:32.107 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_37 2023-12-10T19:17:32.107 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:17:32.107 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:17:32.107 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:32.107 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_37 2023-12-10T19:17:32.107 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster2-mirror-image_37.status 2023-12-10T19:17:32.107 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2-mirror-image_37.status 2023-12-10T19:17:32.108 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:17:32.108 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_37.status 2023-12-10T19:17:32.109 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-image_37.status 2023-12-10T19:17:32.109 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-image_37.status 2023-12-10T19:17:32.109 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:17:32.109 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_37.status 2023-12-10T19:17:32.109 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mirror_image_snapshot cluster2 mirror image_37 2023-12-10T19:17:32.109 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:17:32.109 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:32.109 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_37 2023-12-10T19:17:32.109 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_37 2023-12-10T19:17:32.825 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Snapshot ID: 191 2023-12-10T19:17:32.837 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_37 /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_37.status 2023-12-10T19:17:32.837 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:17:32.837 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:32.837 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_37 2023-12-10T19:17:32.838 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_37.status 2023-12-10T19:17:32.838 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_37 --format xml 2023-12-10T19:17:32.838 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:17:32.893 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-10T19:17:32.895 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snapshot_id=191 2023-12-10T19:17:32.895 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:17:32.895 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.2 2023-12-10T19:17:33.096 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_37.status 2023-12-10T19:17:33.097 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:33.097 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:33.097 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_37 2023-12-10T19:17:33.097 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_37.status 2023-12-10T19:17:33.097 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2023-12-10T19:17:33.097 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:17:33.164 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:17:33.166 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=104 2023-12-10T19:17:33.166 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 191 = 104 2023-12-10T19:17:33.166 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.4 2023-12-10T19:17:33.568 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_37.status 2023-12-10T19:17:33.568 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:33.568 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:33.568 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_37 2023-12-10T19:17:33.568 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_37.status 2023-12-10T19:17:33.568 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2023-12-10T19:17:33.568 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:17:33.640 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:17:33.643 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=104 2023-12-10T19:17:33.643 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 191 = 104 2023-12-10T19:17:33.643 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.8 2023-12-10T19:17:34.444 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_37.status 2023-12-10T19:17:34.444 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:34.444 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:34.445 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_37 2023-12-10T19:17:34.445 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_37.status 2023-12-10T19:17:34.445 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2023-12-10T19:17:34.445 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:17:34.513 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:17:34.513 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:17:34.515 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:17:34.515 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:17:34.515 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:17:34.515 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:17:34.515 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 191 = 2023-12-10T19:17:34.515 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1.6 2023-12-10T19:17:36.116 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_37.status 2023-12-10T19:17:36.116 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:36.116 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:36.116 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_37 2023-12-10T19:17:36.116 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_37.status 2023-12-10T19:17:36.117 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2023-12-10T19:17:36.117 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:17:36.183 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:17:36.187 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=191 2023-12-10T19:17:36.187 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 191 = 191 2023-12-10T19:17:36.187 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:17:36.187 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snap_present cluster1 mirror image_37 snap 2023-12-10T19:17:36.187 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:36.187 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:36.187 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_37 2023-12-10T19:17:36.188 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:17:36.188 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:17:36.188 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:17:36.868 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2023-12-10T19:17:36.868 INFO:teuthology.orchestra.run:waiting for 600 2023-12-10T19:17:36.869 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2023-12-10T19:17:36.869 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-12-10T19:17:36.869 DEBUG:teuthology.orchestra.run.smithi076:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-12-10T19:17:36.872 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-12-10T19:17:36.873 INFO:tasks.rbd_mirror_thrash:waiting for 77 secs before thrashing 2023-12-10T19:17:37.187 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 info mirror/image_37@snap 2023-12-10T19:17:37.247 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_37': 2023-12-10T19:17:37.247 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:17:37.247 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:17:37.247 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 2 2023-12-10T19:17:37.247 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 10c0f4d2efa8 2023-12-10T19:17:37.247 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.10c0f4d2efa8 2023-12-10T19:17:37.248 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:17:37.248 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering, non-primary 2023-12-10T19:17:37.248 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:17:37.248 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:17:37.248 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:09:01 2023 2023-12-10T19:17:37.248 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:09:01 2023 2023-12-10T19:17:37.248 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:09:01 2023 2023-12-10T19:17:37.248 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: protected: False 2023-12-10T19:17:37.248 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:17:37.248 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:17:37.248 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: a4d8c861-aa4e-4c85-9970-1f362ca1c6b6 2023-12-10T19:17:37.248 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: false 2023-12-10T19:17:37.252 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:17:37.252 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror image_37 snap 2023-12-10T19:17:37.252 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:37.252 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_37 2023-12-10T19:17:37.252 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:17:37.252 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:17:37.252 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_37.export 2023-12-10T19:17:37.252 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_37.export 2023-12-10T19:17:37.253 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_37.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_37.export 2023-12-10T19:17:37.253 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export image_37@snap /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_37.export 2023-12-10T19:17:37.480 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:17:37.489 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export image_37@snap /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_37.export 2023-12-10T19:17:37.831 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:17:37.844 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_37.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_37.export 2023-12-10T19:17:37.922 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_37.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_37.export 2023-12-10T19:17:37.961 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:17:37.961 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_38 2023-12-10T19:17:37.961 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror image_38 snap 2023-12-10T19:17:37.961 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:17:37.961 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:37.961 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_38 2023-12-10T19:17:37.961 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:17:37.961 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create image_38@snap --debug-rbd=20 --debug-journaler=20 2023-12-10T19:17:38.755 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_started cluster1 mirror image_38 2023-12-10T19:17:38.755 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:38.755 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:38.755 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_38 2023-12-10T19:17:38.755 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_state cluster1 mirror image_38 started 2023-12-10T19:17:38.755 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:38.755 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:38.756 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_38 2023-12-10T19:17:38.756 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=started 2023-12-10T19:17:38.756 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:17:38.756 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:17:39.757 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_replay_state cluster1 mirror image_38 started 2023-12-10T19:17:39.758 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:39.758 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:39.758 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_38 2023-12-10T19:17:39.758 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=started 2023-12-10T19:17:39.758 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_result 2023-12-10T19:17:39.758 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=stopped 2023-12-10T19:17:39.758 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_38 2023-12-10T19:17:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_instance=cluster1 2023-12-10T19:17:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:17:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ + local cluster=cluster1 2023-12-10T19:17:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance=cluster1 2023-12-10T19:17:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loop_instance 2023-12-10T19:17:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:grep+ -i statesleep 2023-12-10T19:17:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 0 2023-12-10T19:17:39.762 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T19:17:39.762 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T19:17:39.762 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_38 2023-12-10T19:17:39.762 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T19:17:39.762 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:17:39.762 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:17:39.762 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T19:17:39.762 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:17:39.762 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:17:39.763 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:17:39.763 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:17:39.763 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:17:39.763 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:17:39.763 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:17:39.763 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:17:39.763 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:17:39.763 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:17:39.763 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T19:17:39.763 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T19:17:39.763 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:39.764 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:17:39.764 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T19:17:39.764 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:17:39.764 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:17:39.764 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:17:39.764 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:17:39.764 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:17:39.764 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:17:39.765 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:17:39.765 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:17:39.765 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:17:39.765 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:17:39.765 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T19:17:39.781 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:17:39.781 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:17:39.782 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:17:39.782 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_38 2023-12-10T19:17:39.900 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2023-12-10T19:17:39.901 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:17:39.901 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:17:39.903 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_38 2023-12-10T19:17:39.904 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:1 2023-12-10T19:17:39.904 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:17:39.904 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:17:39.904 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-10T19:17:39.904 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:17:39.904 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:17:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:17:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:17:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:17:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:17:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:17:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:17:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:17:39.906 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:17:39.906 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-10T19:17:39.906 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:1 2023-12-10T19:17:39.906 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:39.906 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:17:39.906 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-10T19:17:39.907 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:17:39.907 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:17:39.907 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:17:39.907 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:17:39.907 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:17:39.907 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:17:39.907 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:17:39.908 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:17:39.908 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:17:39.908 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:17:39.908 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-10T19:17:39.926 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:17:39.926 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:17:39.926 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:17:39.926 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_38 2023-12-10T19:17:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2023-12-10T19:17:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:17:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:17:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_38 2023-12-10T19:17:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:2 2023-12-10T19:17:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:17:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:17:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-10T19:17:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:17:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:17:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:17:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:17:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:17:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:17:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:17:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:17:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:17:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:17:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-10T19:17:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:2 2023-12-10T19:17:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:17:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-10T19:17:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:17:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:17:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:17:40.044 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:17:40.044 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:17:40.044 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:17:40.044 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:17:40.044 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:17:40.044 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:17:40.044 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:17:40.044 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-10T19:17:40.067 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:17:40.068 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:17:40.068 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:17:40.068 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_38 2023-12-10T19:17:40.199 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:17:40.200 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ status_result= "state": "Replaying" 2023-12-10T19:17:40.200 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo "state": "Replaying" 2023-12-10T19:17:40.200 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep -i Replaying 2023-12-10T19:17:40.201 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "state": "Replaying" 2023-12-10T19:17:40.203 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=started 2023-12-10T19:17:40.203 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test started = started 2023-12-10T19:17:40.203 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:17:40.203 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_38 2023-12-10T19:17:40.203 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:17:40.203 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:17:40.203 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:40.203 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_38 2023-12-10T19:17:40.203 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = journal ] 2023-12-10T19:17:40.204 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = snapshot ] 2023-12-10T19:17:40.204 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_38 2023-12-10T19:17:40.204 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:17:40.204 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:17:40.204 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:40.204 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_38 2023-12-10T19:17:40.204 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster2-mirror-image_38.status 2023-12-10T19:17:40.204 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2-mirror-image_38.status 2023-12-10T19:17:40.204 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:17:40.205 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_38.status 2023-12-10T19:17:40.205 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-image_38.status 2023-12-10T19:17:40.205 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-image_38.status 2023-12-10T19:17:40.205 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:17:40.205 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_38.status 2023-12-10T19:17:40.206 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mirror_image_snapshot cluster2 mirror image_38 2023-12-10T19:17:40.206 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:17:40.206 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:40.206 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_38 2023-12-10T19:17:40.206 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_38 2023-12-10T19:17:40.770 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Snapshot ID: 193 2023-12-10T19:17:40.785 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_38 /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_38.status 2023-12-10T19:17:40.785 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:17:40.785 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:40.785 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_38 2023-12-10T19:17:40.785 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_38.status 2023-12-10T19:17:40.785 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ + xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:17:40.785 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd --cluster cluster2 snap list --all mirror/image_38 --format xml 2023-12-10T19:17:40.841 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-10T19:17:40.844 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snapshot_id=193 2023-12-10T19:17:40.844 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:17:40.844 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.2 2023-12-10T19:17:41.045 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_38.status 2023-12-10T19:17:41.045 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:41.045 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:41.045 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_38 2023-12-10T19:17:41.045 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_38.status 2023-12-10T19:17:41.045 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2023-12-10T19:17:41.046 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:17:41.125 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:17:41.127 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=105 2023-12-10T19:17:41.127 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 193 = 105 2023-12-10T19:17:41.128 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.4 2023-12-10T19:17:41.529 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_38.status 2023-12-10T19:17:41.529 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:41.529 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:41.529 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_38 2023-12-10T19:17:41.529 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_38.status 2023-12-10T19:17:41.529 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2023-12-10T19:17:41.529 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:17:41.603 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:17:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=105 2023-12-10T19:17:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 193 = 105 2023-12-10T19:17:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.8 2023-12-10T19:17:42.408 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_38.status 2023-12-10T19:17:42.408 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:42.408 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:42.408 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_38 2023-12-10T19:17:42.408 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_38.status 2023-12-10T19:17:42.409 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2023-12-10T19:17:42.409 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:17:42.482 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:17:42.482 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:17:42.484 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:17:42.484 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:17:42.484 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:17:42.485 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:17:42.485 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 193 = 2023-12-10T19:17:42.485 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1.6 2023-12-10T19:17:44.086 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_38.status 2023-12-10T19:17:44.086 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:44.086 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:44.086 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_38 2023-12-10T19:17:44.086 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_38.status 2023-12-10T19:17:44.087 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2023-12-10T19:17:44.087 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:17:44.154 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:17:44.156 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=193 2023-12-10T19:17:44.156 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 193 = 193 2023-12-10T19:17:44.156 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:17:44.156 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snap_present cluster1 mirror image_38 snap 2023-12-10T19:17:44.156 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:44.156 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:44.157 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_38 2023-12-10T19:17:44.157 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:17:44.157 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:17:44.157 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:17:45.157 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 info mirror/image_38@snap 2023-12-10T19:17:45.217 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_38': 2023-12-10T19:17:45.218 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:17:45.218 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:17:45.218 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 2 2023-12-10T19:17:45.218 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 10e8b70d3385 2023-12-10T19:17:45.218 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.10e8b70d3385 2023-12-10T19:17:45.218 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:17:45.218 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering, non-primary 2023-12-10T19:17:45.218 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:17:45.218 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:17:45.218 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:09:16 2023 2023-12-10T19:17:45.218 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:09:16 2023 2023-12-10T19:17:45.218 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:09:16 2023 2023-12-10T19:17:45.218 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: protected: False 2023-12-10T19:17:45.218 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:17:45.219 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:17:45.219 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: 82e2c879-d2a6-4227-b47b-c52a54933f28 2023-12-10T19:17:45.219 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: false 2023-12-10T19:17:45.222 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:17:45.222 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror image_38 snap 2023-12-10T19:17:45.222 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:45.223 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_38 2023-12-10T19:17:45.223 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:17:45.223 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:17:45.223 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_38.export 2023-12-10T19:17:45.223 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_38.export 2023-12-10T19:17:45.223 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_38.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_38.export 2023-12-10T19:17:45.223 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export image_38@snap /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_38.export 2023-12-10T19:17:45.418 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:17:45.427 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export image_38@snap /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_38.export 2023-12-10T19:17:45.687 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:17:45.700 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_38.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_38.export 2023-12-10T19:17:45.777 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_38.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_38.export 2023-12-10T19:17:45.817 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:17:45.817 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_39 2023-12-10T19:17:45.817 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror image_39 snap 2023-12-10T19:17:45.817 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:17:45.817 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:45.817 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_39 2023-12-10T19:17:45.817 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:17:45.817 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create image_39@snap --debug-rbd=20 --debug-journaler=20 2023-12-10T19:17:46.435 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_started cluster1 mirror image_39 2023-12-10T19:17:46.435 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:46.435 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:46.435 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_39 2023-12-10T19:17:46.435 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_state cluster1 mirror image_39 started 2023-12-10T19:17:46.435 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:46.435 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:46.435 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_39 2023-12-10T19:17:46.435 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=started 2023-12-10T19:17:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:17:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:17:47.437 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_replay_state cluster1 mirror image_39 started 2023-12-10T19:17:47.438 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:47.438 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:47.438 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_39 2023-12-10T19:17:47.438 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=started 2023-12-10T19:17:47.438 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_result 2023-12-10T19:17:47.438 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=stopped 2023-12-10T19:17:47.438 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_39 2023-12-10T19:17:47.439 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_instance=cluster1 2023-12-10T19:17:47.439 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:17:47.439 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ + local cluster=cluster1 2023-12-10T19:17:47.439 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:grep+ -ilocal state instance=cluster1 2023-12-10T19:17:47.439 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:17:47.439 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loop_instance 2023-12-10T19:17:47.439 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:17:47.442 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T19:17:47.442 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T19:17:47.442 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_39 2023-12-10T19:17:47.442 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T19:17:47.442 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:17:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:17:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T19:17:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:17:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:17:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:17:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:17:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:17:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:17:47.444 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:17:47.444 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:17:47.444 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:17:47.444 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:17:47.444 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T19:17:47.444 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T19:17:47.444 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:47.444 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:17:47.444 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T19:17:47.444 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:17:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:17:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:17:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:17:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:17:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:17:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:17:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:17:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:17:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:17:47.446 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T19:17:47.465 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:17:47.465 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:17:47.465 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:17:47.465 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_39 2023-12-10T19:17:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2023-12-10T19:17:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:17:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:17:47.578 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_39 2023-12-10T19:17:47.578 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:1 2023-12-10T19:17:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:17:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:17:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-10T19:17:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:17:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:17:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:17:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:17:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:17:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:17:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:17:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:17:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:17:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:17:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-10T19:17:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:1 2023-12-10T19:17:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:17:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-10T19:17:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:17:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:17:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:17:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:17:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:17:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:17:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:17:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:17:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:17:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:17:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-10T19:17:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:17:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:17:47.599 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:17:47.599 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_39 2023-12-10T19:17:47.710 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2023-12-10T19:17:47.710 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:17:47.711 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:17:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_39 2023-12-10T19:17:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:2 2023-12-10T19:17:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:17:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:17:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-10T19:17:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:17:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:17:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:17:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:17:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:17:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:17:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:17:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:17:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:17:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:17:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-10T19:17:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:2 2023-12-10T19:17:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:17:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-10T19:17:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:17:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:17:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:17:47.715 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:17:47.715 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:17:47.715 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:17:47.715 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:17:47.715 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:17:47.715 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:17:47.715 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:17:47.715 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-10T19:17:47.737 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:17:47.737 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:17:47.737 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:17:47.737 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_39 2023-12-10T19:17:47.875 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:17:47.875 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ status_result= "state": "Replaying" 2023-12-10T19:17:47.875 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo "state": "Replaying" 2023-12-10T19:17:47.875 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep -i Replaying 2023-12-10T19:17:47.876 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "state": "Replaying" 2023-12-10T19:17:47.877 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=started 2023-12-10T19:17:47.877 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test started = started 2023-12-10T19:17:47.877 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:17:47.877 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_39 2023-12-10T19:17:47.877 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:17:47.877 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:17:47.877 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:47.877 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_39 2023-12-10T19:17:47.877 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = journal ] 2023-12-10T19:17:47.877 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = snapshot ] 2023-12-10T19:17:47.877 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_39 2023-12-10T19:17:47.878 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:17:47.878 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:17:47.878 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:47.878 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_39 2023-12-10T19:17:47.878 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster2-mirror-image_39.status 2023-12-10T19:17:47.878 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2-mirror-image_39.status 2023-12-10T19:17:47.878 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:17:47.879 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_39.status 2023-12-10T19:17:47.879 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-image_39.status 2023-12-10T19:17:47.879 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-image_39.status 2023-12-10T19:17:47.879 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:17:47.879 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_39.status 2023-12-10T19:17:47.879 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mirror_image_snapshot cluster2 mirror image_39 2023-12-10T19:17:47.879 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:17:47.879 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:47.880 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_39 2023-12-10T19:17:47.880 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_39 2023-12-10T19:17:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Snapshot ID: 195 2023-12-10T19:17:48.470 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_39 /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_39.status 2023-12-10T19:17:48.470 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:17:48.470 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:48.470 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_39 2023-12-10T19:17:48.470 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_39.status 2023-12-10T19:17:48.470 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_39 --format xml 2023-12-10T19:17:48.470 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:17:48.524 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-10T19:17:48.527 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snapshot_id=195 2023-12-10T19:17:48.527 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:17:48.527 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.2 2023-12-10T19:17:48.728 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_39.status 2023-12-10T19:17:48.728 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:48.728 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:48.728 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_39 2023-12-10T19:17:48.728 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_39.status 2023-12-10T19:17:48.728 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2023-12-10T19:17:48.729 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:17:48.794 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:17:48.795 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:17:48.796 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:17:48.796 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:17:48.796 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:17:48.797 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:17:48.797 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 195 = 2023-12-10T19:17:48.797 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.4 2023-12-10T19:17:49.198 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_39.status 2023-12-10T19:17:49.198 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:49.199 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:49.199 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_39 2023-12-10T19:17:49.199 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_39.status 2023-12-10T19:17:49.199 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2023-12-10T19:17:49.199 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:17:49.269 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:17:49.269 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:17:49.271 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:17:49.271 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:17:49.271 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:17:49.271 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:17:49.271 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 195 = 2023-12-10T19:17:49.271 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.8 2023-12-10T19:17:50.072 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_39.status 2023-12-10T19:17:50.073 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:50.073 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:50.073 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_39 2023-12-10T19:17:50.073 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_39.status 2023-12-10T19:17:50.073 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2023-12-10T19:17:50.073 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:17:50.139 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:17:50.142 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=195 2023-12-10T19:17:50.142 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 195 = 195 2023-12-10T19:17:50.142 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:17:50.142 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snap_present cluster1 mirror image_39 snap 2023-12-10T19:17:50.142 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:50.142 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:50.142 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_39 2023-12-10T19:17:50.142 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:17:50.143 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:17:50.143 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:17:51.143 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 info mirror/image_39@snap 2023-12-10T19:17:51.203 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_39': 2023-12-10T19:17:51.203 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:17:51.203 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:17:51.203 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 2 2023-12-10T19:17:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 128b20d7ef27 2023-12-10T19:17:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.128b20d7ef27 2023-12-10T19:17:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:17:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering, non-primary 2023-12-10T19:17:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:17:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:17:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:09:28 2023 2023-12-10T19:17:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:09:28 2023 2023-12-10T19:17:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:09:28 2023 2023-12-10T19:17:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: protected: False 2023-12-10T19:17:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:17:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:17:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: b649fd8b-2f2a-4f7d-94c5-3e9e6a6f4c99 2023-12-10T19:17:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: false 2023-12-10T19:17:51.208 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:17:51.208 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror image_39 snap 2023-12-10T19:17:51.208 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:51.208 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_39 2023-12-10T19:17:51.208 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:17:51.208 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:17:51.208 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_39.export 2023-12-10T19:17:51.208 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_39.export 2023-12-10T19:17:51.208 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_39.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_39.export 2023-12-10T19:17:51.209 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export image_39@snap /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_39.export 2023-12-10T19:17:51.417 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:17:51.426 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export image_39@snap /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_39.export 2023-12-10T19:17:51.688 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:17:51.701 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_39.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_39.export 2023-12-10T19:17:51.778 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_39.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_39.export 2023-12-10T19:17:51.818 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:17:51.818 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_40 2023-12-10T19:17:51.818 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror image_40 snap 2023-12-10T19:17:51.818 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:17:51.818 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:51.818 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_40 2023-12-10T19:17:51.818 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:17:51.819 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create image_40@snap --debug-rbd=20 --debug-journaler=20 2023-12-10T19:17:52.768 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_started cluster1 mirror image_40 2023-12-10T19:17:52.769 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:52.769 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:52.769 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_40 2023-12-10T19:17:52.769 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_state cluster1 mirror image_40 started 2023-12-10T19:17:52.769 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:52.769 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:52.769 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_40 2023-12-10T19:17:52.769 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=started 2023-12-10T19:17:52.769 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:17:52.769 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:17:53.769 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_replay_state cluster1 mirror image_40 started 2023-12-10T19:17:53.770 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:53.770 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:53.770 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_40 2023-12-10T19:17:53.770 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=started 2023-12-10T19:17:53.770 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_result 2023-12-10T19:17:53.770 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=stopped 2023-12-10T19:17:53.771 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_40 2023-12-10T19:17:53.771 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_instance=cluster1 2023-12-10T19:17:53.771 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:17:53.771 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:53.771 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance=cluster1 2023-12-10T19:17:53.771 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loop_instance 2023-12-10T19:17:53.771 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:17:53.771 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep -i state 2023-12-10T19:17:53.772 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T19:17:53.772 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T19:17:53.773 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_40 2023-12-10T19:17:53.773 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T19:17:53.773 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:17:53.773 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:17:53.773 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T19:17:53.773 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:17:53.773 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:17:53.773 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:17:53.773 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:17:53.773 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:17:53.773 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:17:53.774 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:17:53.774 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:17:53.774 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:17:53.774 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:17:53.774 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T19:17:53.774 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T19:17:53.774 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:53.774 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:17:53.774 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T19:17:53.774 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:17:53.774 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:17:53.774 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:17:53.774 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:17:53.774 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:17:53.775 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:17:53.775 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:17:53.775 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:17:53.775 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:17:53.775 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:17:53.775 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T19:17:53.794 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:17:53.794 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:17:53.794 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:17:53.794 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_40 2023-12-10T19:17:53.909 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2023-12-10T19:17:53.910 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:17:53.910 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:17:53.912 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_40 2023-12-10T19:17:53.912 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:1 2023-12-10T19:17:53.912 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:17:53.912 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:17:53.912 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-10T19:17:53.912 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:17:53.912 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:17:53.912 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:17:53.912 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:17:53.912 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:17:53.912 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:17:53.913 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:17:53.913 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:17:53.913 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:17:53.913 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:17:53.913 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-10T19:17:53.913 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:1 2023-12-10T19:17:53.913 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:53.913 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:17:53.913 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-10T19:17:53.913 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:17:53.913 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:17:53.913 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:17:53.913 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:17:53.913 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:17:53.914 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:17:53.914 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:17:53.914 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:17:53.914 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:17:53.914 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:17:53.914 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-10T19:17:53.937 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:17:53.937 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:17:53.937 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:17:53.937 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_40 2023-12-10T19:17:54.070 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:17:54.071 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ status_result= "state": "Replaying" 2023-12-10T19:17:54.071 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo "state": "Replaying" 2023-12-10T19:17:54.071 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep -i Replaying 2023-12-10T19:17:54.072 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "state": "Replaying" 2023-12-10T19:17:54.073 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=started 2023-12-10T19:17:54.073 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test started = started 2023-12-10T19:17:54.073 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:17:54.073 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_40 2023-12-10T19:17:54.074 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:17:54.074 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:17:54.074 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:54.074 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_40 2023-12-10T19:17:54.074 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = journal ] 2023-12-10T19:17:54.074 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = snapshot ] 2023-12-10T19:17:54.074 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_40 2023-12-10T19:17:54.074 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:17:54.074 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:17:54.074 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:54.074 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_40 2023-12-10T19:17:54.074 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster2-mirror-image_40.status 2023-12-10T19:17:54.074 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2-mirror-image_40.status 2023-12-10T19:17:54.074 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:17:54.075 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_40.status 2023-12-10T19:17:54.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-image_40.status 2023-12-10T19:17:54.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-image_40.status 2023-12-10T19:17:54.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:17:54.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_40.status 2023-12-10T19:17:54.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mirror_image_snapshot cluster2 mirror image_40 2023-12-10T19:17:54.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:17:54.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:54.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_40 2023-12-10T19:17:54.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_40 2023-12-10T19:17:54.790 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Snapshot ID: 197 2023-12-10T19:17:54.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_40 /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_40.status 2023-12-10T19:17:54.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:17:54.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:54.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_40 2023-12-10T19:17:54.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_40.status 2023-12-10T19:17:54.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_40 --format xml 2023-12-10T19:17:54.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:17:54.859 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-10T19:17:54.861 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snapshot_id=197 2023-12-10T19:17:54.861 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:17:54.862 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.2 2023-12-10T19:17:55.062 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_40.status 2023-12-10T19:17:55.063 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:55.063 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:55.063 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_40 2023-12-10T19:17:55.063 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_40.status 2023-12-10T19:17:55.063 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2023-12-10T19:17:55.063 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:17:55.138 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:17:55.141 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=107 2023-12-10T19:17:55.141 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 197 = 107 2023-12-10T19:17:55.141 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.4 2023-12-10T19:17:55.542 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_40.status 2023-12-10T19:17:55.543 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:55.543 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:55.543 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_40 2023-12-10T19:17:55.543 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_40.status 2023-12-10T19:17:55.543 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2023-12-10T19:17:55.543 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:17:55.609 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:17:55.611 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=107 2023-12-10T19:17:55.612 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 197 = 107 2023-12-10T19:17:55.612 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.8 2023-12-10T19:17:56.413 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_40.status 2023-12-10T19:17:56.413 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:56.413 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:56.413 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_40 2023-12-10T19:17:56.413 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_40.status 2023-12-10T19:17:56.414 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2023-12-10T19:17:56.414 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:17:56.485 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:17:56.485 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:17:56.487 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:17:56.487 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:17:56.487 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:17:56.488 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:17:56.488 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 197 = 2023-12-10T19:17:56.488 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1.6 2023-12-10T19:17:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_40.status 2023-12-10T19:17:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_40 2023-12-10T19:17:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_40.status 2023-12-10T19:17:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2023-12-10T19:17:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:17:58.155 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:17:58.158 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=197 2023-12-10T19:17:58.158 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 197 = 197 2023-12-10T19:17:58.158 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:17:58.158 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snap_present cluster1 mirror image_40 snap 2023-12-10T19:17:58.158 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:17:58.158 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:58.158 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_40 2023-12-10T19:17:58.158 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:17:58.159 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:17:58.159 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:17:59.159 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 info mirror/image_40@snap 2023-12-10T19:17:59.222 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_40': 2023-12-10T19:17:59.222 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:17:59.222 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:17:59.222 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 2 2023-12-10T19:17:59.222 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 12a8271a3865 2023-12-10T19:17:59.222 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.12a8271a3865 2023-12-10T19:17:59.222 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:17:59.222 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering, non-primary 2023-12-10T19:17:59.222 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:17:59.222 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:17:59.222 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:09:39 2023 2023-12-10T19:17:59.222 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:09:39 2023 2023-12-10T19:17:59.222 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:09:39 2023 2023-12-10T19:17:59.223 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: protected: False 2023-12-10T19:17:59.223 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:17:59.223 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:17:59.223 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: 36d7ae4a-9a61-4f12-8a89-e0b759776b33 2023-12-10T19:17:59.223 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: false 2023-12-10T19:17:59.226 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:17:59.226 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror image_40 snap 2023-12-10T19:17:59.226 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:59.226 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_40 2023-12-10T19:17:59.226 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:17:59.226 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:17:59.226 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_40.export 2023-12-10T19:17:59.227 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_40.export 2023-12-10T19:17:59.227 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_40.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_40.export 2023-12-10T19:17:59.227 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export image_40@snap /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_40.export 2023-12-10T19:17:59.428 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:17:59.436 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export image_40@snap /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_40.export 2023-12-10T19:17:59.786 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:17:59.801 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_40.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_40.export 2023-12-10T19:17:59.879 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_40.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_40.export 2023-12-10T19:17:59.919 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:17:59.919 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_41 2023-12-10T19:17:59.919 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror image_41 snap 2023-12-10T19:17:59.919 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:17:59.919 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:17:59.920 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_41 2023-12-10T19:17:59.920 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:17:59.920 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create image_41@snap --debug-rbd=20 --debug-journaler=20 2023-12-10T19:18:00.793 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_started cluster1 mirror image_41 2023-12-10T19:18:00.793 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:00.794 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:00.794 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_41 2023-12-10T19:18:00.794 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_state cluster1 mirror image_41 started 2023-12-10T19:18:00.794 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:00.794 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:00.794 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_41 2023-12-10T19:18:00.794 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=started 2023-12-10T19:18:00.794 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:18:00.794 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:18:01.794 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_replay_state cluster1 mirror image_41 started 2023-12-10T19:18:01.794 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:01.794 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:01.794 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_41 2023-12-10T19:18:01.795 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=started 2023-12-10T19:18:01.795 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_result 2023-12-10T19:18:01.795 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=stopped 2023-12-10T19:18:01.795 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_41 2023-12-10T19:18:01.795 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_instance=cluster1 2023-12-10T19:18:01.795 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:18:01.796 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:01.796 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance=cluster1 2023-12-10T19:18:01.796 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loop_instance 2023-12-10T19:18:01.796 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:18:01.796 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep -i state 2023-12-10T19:18:01.797 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T19:18:01.797 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T19:18:01.797 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_41 2023-12-10T19:18:01.797 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T19:18:01.797 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:18:01.798 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:18:01.798 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T19:18:01.798 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:18:01.798 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:18:01.798 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:18:01.798 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:18:01.798 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:18:01.798 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:18:01.798 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:18:01.798 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:18:01.798 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:18:01.798 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:18:01.798 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T19:18:01.798 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T19:18:01.798 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:01.798 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:18:01.799 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T19:18:01.799 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:18:01.799 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:18:01.799 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:18:01.799 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:18:01.799 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:18:01.799 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:18:01.799 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:18:01.799 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:18:01.799 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:18:01.799 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:18:01.799 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T19:18:01.820 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:18:01.820 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:18:01.820 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:18:01.821 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_41 2023-12-10T19:18:01.932 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2023-12-10T19:18:01.932 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:18:01.932 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:18:01.934 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_41 2023-12-10T19:18:01.934 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:1 2023-12-10T19:18:01.934 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:18:01.934 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:18:01.934 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-10T19:18:01.934 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:18:01.934 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:18:01.934 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:18:01.934 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:18:01.934 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:18:01.934 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:18:01.934 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:18:01.934 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:18:01.934 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:18:01.935 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:18:01.936 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-10T19:18:01.936 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:1 2023-12-10T19:18:01.936 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:01.936 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:18:01.936 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-10T19:18:01.936 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:18:01.936 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:18:01.936 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:18:01.936 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:18:01.936 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:18:01.937 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:18:01.937 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:18:01.937 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:18:01.937 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:18:01.937 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:18:01.937 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-10T19:18:01.954 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:18:01.954 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:18:01.954 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:18:01.955 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_41 2023-12-10T19:18:02.063 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2023-12-10T19:18:02.063 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:18:02.063 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:18:02.065 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_41 2023-12-10T19:18:02.065 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:2 2023-12-10T19:18:02.065 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:18:02.065 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:18:02.065 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-10T19:18:02.065 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:18:02.065 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:18:02.065 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:18:02.065 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:18:02.066 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:18:02.066 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:18:02.066 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:18:02.066 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:18:02.066 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:18:02.066 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:18:02.067 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-10T19:18:02.067 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:2 2023-12-10T19:18:02.067 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:02.067 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:18:02.067 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-10T19:18:02.067 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:18:02.067 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:18:02.067 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:18:02.067 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:18:02.067 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:18:02.067 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:18:02.067 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:18:02.068 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:18:02.068 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:18:02.068 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:18:02.068 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-10T19:18:02.089 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:18:02.089 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:18:02.089 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:18:02.089 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_41 2023-12-10T19:18:02.208 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:18:02.209 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ status_result= "state": "Replaying" 2023-12-10T19:18:02.209 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo "state": "Replaying" 2023-12-10T19:18:02.210 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep -i Replaying 2023-12-10T19:18:02.211 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "state": "Replaying" 2023-12-10T19:18:02.213 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=started 2023-12-10T19:18:02.213 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test started = started 2023-12-10T19:18:02.213 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:18:02.213 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_41 2023-12-10T19:18:02.213 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:18:02.213 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:18:02.213 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:02.213 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_41 2023-12-10T19:18:02.213 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = journal ] 2023-12-10T19:18:02.213 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = snapshot ] 2023-12-10T19:18:02.213 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_41 2023-12-10T19:18:02.213 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:18:02.213 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:18:02.214 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:02.214 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_41 2023-12-10T19:18:02.214 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster2-mirror-image_41.status 2023-12-10T19:18:02.214 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2-mirror-image_41.status 2023-12-10T19:18:02.214 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:18:02.215 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_41.status 2023-12-10T19:18:02.215 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-image_41.status 2023-12-10T19:18:02.215 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-image_41.status 2023-12-10T19:18:02.215 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:18:02.215 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_41.status 2023-12-10T19:18:02.215 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mirror_image_snapshot cluster2 mirror image_41 2023-12-10T19:18:02.216 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:18:02.216 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:02.216 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_41 2023-12-10T19:18:02.216 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_41 2023-12-10T19:18:02.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Snapshot ID: 199 2023-12-10T19:18:02.826 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_41 /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_41.status 2023-12-10T19:18:02.826 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:18:02.826 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:02.826 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_41 2023-12-10T19:18:02.826 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_41.status 2023-12-10T19:18:02.826 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_41 --format xml 2023-12-10T19:18:02.826 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:18:02.880 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-10T19:18:02.882 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snapshot_id=199 2023-12-10T19:18:02.883 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:18:02.883 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.2 2023-12-10T19:18:03.084 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_41.status 2023-12-10T19:18:03.084 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:03.084 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:03.084 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_41 2023-12-10T19:18:03.084 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_41.status 2023-12-10T19:18:03.085 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2023-12-10T19:18:03.085 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:18:03.152 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:18:03.155 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=108 2023-12-10T19:18:03.155 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 199 = 108 2023-12-10T19:18:03.155 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.4 2023-12-10T19:18:03.556 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_41.status 2023-12-10T19:18:03.556 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:03.556 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:03.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_41 2023-12-10T19:18:03.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_41.status 2023-12-10T19:18:03.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2023-12-10T19:18:03.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:18:03.623 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:18:03.626 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=108 2023-12-10T19:18:03.626 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 199 = 108 2023-12-10T19:18:03.626 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.8 2023-12-10T19:18:04.428 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_41.status 2023-12-10T19:18:04.428 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:04.428 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:04.428 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_41 2023-12-10T19:18:04.428 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_41.status 2023-12-10T19:18:04.428 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2023-12-10T19:18:04.428 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:18:04.495 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:18:04.495 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:18:04.497 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:18:04.498 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:18:04.498 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:18:04.498 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:18:04.498 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 199 = 2023-12-10T19:18:04.498 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1.6 2023-12-10T19:18:06.099 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_41.status 2023-12-10T19:18:06.099 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:06.099 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:06.100 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_41 2023-12-10T19:18:06.100 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_41.status 2023-12-10T19:18:06.100 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2023-12-10T19:18:06.100 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:18:06.167 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:18:06.170 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=199 2023-12-10T19:18:06.170 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 199 = 199 2023-12-10T19:18:06.170 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:18:06.170 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snap_present cluster1 mirror image_41 snap 2023-12-10T19:18:06.171 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:06.171 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:06.171 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_41 2023-12-10T19:18:06.171 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:18:06.171 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:18:06.171 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:18:07.171 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 info mirror/image_41@snap 2023-12-10T19:18:07.232 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_41': 2023-12-10T19:18:07.232 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:18:07.232 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:18:07.232 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 2 2023-12-10T19:18:07.232 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 10c076a5f5f1 2023-12-10T19:18:07.232 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.10c076a5f5f1 2023-12-10T19:18:07.232 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:18:07.232 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering, non-primary 2023-12-10T19:18:07.232 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:18:07.232 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:18:07.232 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:09:51 2023 2023-12-10T19:18:07.233 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:09:51 2023 2023-12-10T19:18:07.233 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:09:51 2023 2023-12-10T19:18:07.233 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: protected: False 2023-12-10T19:18:07.233 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:18:07.233 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:18:07.233 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: 0caf4caf-9ee1-436f-874d-71291f8008fb 2023-12-10T19:18:07.233 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: false 2023-12-10T19:18:07.236 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:18:07.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror image_41 snap 2023-12-10T19:18:07.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:07.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_41 2023-12-10T19:18:07.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:18:07.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:18:07.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_41.export 2023-12-10T19:18:07.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_41.export 2023-12-10T19:18:07.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_41.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_41.export 2023-12-10T19:18:07.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export image_41@snap /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_41.export 2023-12-10T19:18:07.437 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:18:07.445 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export image_41@snap /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_41.export 2023-12-10T19:18:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:18:07.760 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_41.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_41.export 2023-12-10T19:18:07.836 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_41.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_41.export 2023-12-10T19:18:07.876 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:18:07.877 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_42 2023-12-10T19:18:07.877 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror image_42 snap 2023-12-10T19:18:07.877 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:18:07.877 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:07.877 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_42 2023-12-10T19:18:07.877 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:18:07.877 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create image_42@snap --debug-rbd=20 --debug-journaler=20 2023-12-10T19:18:08.768 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_started cluster1 mirror image_42 2023-12-10T19:18:08.768 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:08.769 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:08.769 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_42 2023-12-10T19:18:08.769 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_state cluster1 mirror image_42 started 2023-12-10T19:18:08.769 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:08.769 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:08.769 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_42 2023-12-10T19:18:08.769 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=started 2023-12-10T19:18:08.769 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:18:08.769 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:18:09.769 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_replay_state cluster1 mirror image_42 started 2023-12-10T19:18:09.770 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:09.770 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:09.770 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_42 2023-12-10T19:18:09.770 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=started 2023-12-10T19:18:09.770 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_result 2023-12-10T19:18:09.770 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=stopped 2023-12-10T19:18:09.771 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_42 2023-12-10T19:18:09.771 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ + local cluster_instance=cluster1 2023-12-10T19:18:09.771 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:18:09.771 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:grep -i state 2023-12-10T19:18:09.771 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:09.771 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance=cluster1 2023-12-10T19:18:09.771 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loop_instance 2023-12-10T19:18:09.771 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:18:09.772 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T19:18:09.772 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T19:18:09.773 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_42 2023-12-10T19:18:09.774 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T19:18:09.774 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:18:09.774 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:18:09.774 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T19:18:09.774 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:18:09.774 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:18:09.774 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:18:09.774 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:18:09.774 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:18:09.774 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:18:09.774 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:18:09.774 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:18:09.774 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:18:09.774 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:18:09.774 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T19:18:09.775 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T19:18:09.775 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:09.775 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:18:09.775 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T19:18:09.775 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:18:09.775 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:18:09.775 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:18:09.775 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:18:09.775 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:18:09.775 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:18:09.775 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:18:09.775 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:18:09.775 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:18:09.775 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:18:09.775 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T19:18:09.798 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:18:09.798 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:18:09.798 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:18:09.799 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_42 2023-12-10T19:18:09.931 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:18:09.931 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ status_result= "state": "Replaying" 2023-12-10T19:18:09.931 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo "state": "Replaying" 2023-12-10T19:18:09.931 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep -i Replaying 2023-12-10T19:18:09.932 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "state": "Replaying" 2023-12-10T19:18:09.933 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=started 2023-12-10T19:18:09.933 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test started = started 2023-12-10T19:18:09.933 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:18:09.933 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_42 2023-12-10T19:18:09.933 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:18:09.933 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:18:09.933 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:09.933 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_42 2023-12-10T19:18:09.933 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = journal ] 2023-12-10T19:18:09.933 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = snapshot ] 2023-12-10T19:18:09.933 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_42 2023-12-10T19:18:09.933 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:18:09.933 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:18:09.934 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:09.934 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_42 2023-12-10T19:18:09.934 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster2-mirror-image_42.status 2023-12-10T19:18:09.934 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2-mirror-image_42.status 2023-12-10T19:18:09.934 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:18:09.935 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_42.status 2023-12-10T19:18:09.935 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-image_42.status 2023-12-10T19:18:09.935 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-image_42.status 2023-12-10T19:18:09.935 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:18:09.935 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_42.status 2023-12-10T19:18:09.935 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mirror_image_snapshot cluster2 mirror image_42 2023-12-10T19:18:09.935 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:18:09.935 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:09.935 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_42 2023-12-10T19:18:09.936 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_42 2023-12-10T19:18:10.791 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Snapshot ID: 201 2023-12-10T19:18:10.806 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_42 /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_42.status 2023-12-10T19:18:10.806 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:18:10.806 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:10.806 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_42 2023-12-10T19:18:10.806 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_42.status 2023-12-10T19:18:10.806 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_42 --format xml 2023-12-10T19:18:10.806 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:18:10.862 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-10T19:18:10.864 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snapshot_id=201 2023-12-10T19:18:10.864 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:18:10.864 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.2 2023-12-10T19:18:11.065 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_42.status 2023-12-10T19:18:11.066 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:11.066 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:11.066 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_42 2023-12-10T19:18:11.066 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_42.status 2023-12-10T19:18:11.066 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2023-12-10T19:18:11.066 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:18:11.135 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:18:11.137 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=109 2023-12-10T19:18:11.137 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 201 = 109 2023-12-10T19:18:11.137 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.4 2023-12-10T19:18:11.539 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_42.status 2023-12-10T19:18:11.539 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:11.539 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:11.539 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_42 2023-12-10T19:18:11.539 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_42.status 2023-12-10T19:18:11.539 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2023-12-10T19:18:11.539 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:18:11.609 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:18:11.612 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=109 2023-12-10T19:18:11.612 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 201 = 109 2023-12-10T19:18:11.612 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.8 2023-12-10T19:18:12.413 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_42.status 2023-12-10T19:18:12.413 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:12.414 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:12.414 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_42 2023-12-10T19:18:12.414 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_42.status 2023-12-10T19:18:12.414 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2023-12-10T19:18:12.414 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:18:12.486 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:18:12.486 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:18:12.488 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:18:12.488 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:18:12.488 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:18:12.488 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:18:12.488 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 201 = 2023-12-10T19:18:12.488 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1.6 2023-12-10T19:18:14.090 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_42.status 2023-12-10T19:18:14.090 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:14.090 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:14.090 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_42 2023-12-10T19:18:14.090 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_42.status 2023-12-10T19:18:14.090 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2023-12-10T19:18:14.090 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:18:14.157 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:18:14.161 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=201 2023-12-10T19:18:14.161 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 201 = 201 2023-12-10T19:18:14.161 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:18:14.161 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snap_present cluster1 mirror image_42 snap 2023-12-10T19:18:14.161 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:14.161 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:14.161 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_42 2023-12-10T19:18:14.161 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:18:14.161 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:18:14.161 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:18:15.161 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 info mirror/image_42@snap 2023-12-10T19:18:15.222 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_42': 2023-12-10T19:18:15.222 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:18:15.222 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:18:15.222 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 2 2023-12-10T19:18:15.222 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 10e84b5bf071 2023-12-10T19:18:15.222 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.10e84b5bf071 2023-12-10T19:18:15.223 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:18:15.223 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering, non-primary 2023-12-10T19:18:15.223 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:18:15.223 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:18:15.223 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:10:00 2023 2023-12-10T19:18:15.223 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:10:00 2023 2023-12-10T19:18:15.223 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:10:00 2023 2023-12-10T19:18:15.223 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: protected: False 2023-12-10T19:18:15.223 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:18:15.223 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:18:15.223 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: 472ccd57-dfb6-4e9c-bb2d-657bb53c8829 2023-12-10T19:18:15.223 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: false 2023-12-10T19:18:15.226 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:18:15.226 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror image_42 snap 2023-12-10T19:18:15.227 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:15.227 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_42 2023-12-10T19:18:15.227 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:18:15.227 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:18:15.227 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_42.export 2023-12-10T19:18:15.227 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_42.export 2023-12-10T19:18:15.227 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_42.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_42.export 2023-12-10T19:18:15.227 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export image_42@snap /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_42.export 2023-12-10T19:18:15.428 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:18:15.436 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export image_42@snap /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_42.export 2023-12-10T19:18:15.632 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:18:15.643 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_42.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_42.export 2023-12-10T19:18:15.718 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_42.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_42.export 2023-12-10T19:18:15.757 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:18:15.757 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_43 2023-12-10T19:18:15.757 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror image_43 snap 2023-12-10T19:18:15.758 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:18:15.758 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:15.758 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_43 2023-12-10T19:18:15.758 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:18:15.758 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create image_43@snap --debug-rbd=20 --debug-journaler=20 2023-12-10T19:18:16.442 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_started cluster1 mirror image_43 2023-12-10T19:18:16.442 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:16.443 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:16.443 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_43 2023-12-10T19:18:16.443 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_state cluster1 mirror image_43 started 2023-12-10T19:18:16.443 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:16.443 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:16.443 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_43 2023-12-10T19:18:16.443 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=started 2023-12-10T19:18:16.443 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:18:16.444 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:18:17.443 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_replay_state cluster1 mirror image_43 started 2023-12-10T19:18:17.443 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:17.443 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:17.443 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_43 2023-12-10T19:18:17.443 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=started 2023-12-10T19:18:17.444 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_result 2023-12-10T19:18:17.444 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=stopped 2023-12-10T19:18:17.444 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_43 2023-12-10T19:18:17.444 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_instance=cluster1 2023-12-10T19:18:17.444 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:18:17.444 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:17.445 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance=cluster1 2023-12-10T19:18:17.445 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loop_instance 2023-12-10T19:18:17.445 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:18:17.445 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep -i state 2023-12-10T19:18:17.445 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T19:18:17.445 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T19:18:17.446 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_43 2023-12-10T19:18:17.447 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T19:18:17.447 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:18:17.447 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:18:17.447 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T19:18:17.447 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:18:17.447 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:18:17.447 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:18:17.447 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:18:17.447 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:18:17.447 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:18:17.447 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:18:17.447 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:18:17.447 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:18:17.447 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:18:17.447 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T19:18:17.448 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T19:18:17.448 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:17.448 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:18:17.448 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T19:18:17.448 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:18:17.448 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:18:17.448 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:18:17.448 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:18:17.448 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:18:17.448 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:18:17.448 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:18:17.448 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:18:17.448 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:18:17.448 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:18:17.448 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T19:18:17.474 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:18:17.474 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:18:17.474 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:18:17.474 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_43 2023-12-10T19:18:17.594 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2023-12-10T19:18:17.594 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:18:17.594 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:18:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_43 2023-12-10T19:18:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:1 2023-12-10T19:18:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:18:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:18:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-10T19:18:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:18:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:18:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:18:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:18:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:18:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:18:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:18:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:18:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:18:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:18:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-10T19:18:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:1 2023-12-10T19:18:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:18:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-10T19:18:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:18:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:18:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:18:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:18:17.599 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:18:17.599 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:18:17.599 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:18:17.599 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:18:17.599 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:18:17.599 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:18:17.599 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-10T19:18:17.620 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:18:17.620 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:18:17.620 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:18:17.620 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_43 2023-12-10T19:18:17.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2023-12-10T19:18:17.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:18:17.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:18:17.731 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_43 2023-12-10T19:18:17.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:2 2023-12-10T19:18:17.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:18:17.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:18:17.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-10T19:18:17.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:18:17.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:18:17.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:18:17.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:18:17.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:18:17.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:18:17.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:18:17.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:18:17.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:18:17.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:18:17.733 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-10T19:18:17.734 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:2 2023-12-10T19:18:17.734 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:17.734 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:18:17.734 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-10T19:18:17.734 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:18:17.734 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:18:17.734 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:18:17.734 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:18:17.734 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:18:17.734 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:18:17.734 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:18:17.734 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:18:17.734 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:18:17.734 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:18:17.735 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-10T19:18:17.756 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:18:17.757 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:18:17.757 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:18:17.757 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_43 2023-12-10T19:18:17.891 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:18:17.891 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ status_result= "state": "Replaying" 2023-12-10T19:18:17.892 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo "state": "Replaying" 2023-12-10T19:18:17.892 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep -i Replaying 2023-12-10T19:18:17.893 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "state": "Replaying" 2023-12-10T19:18:17.894 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=started 2023-12-10T19:18:17.894 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test started = started 2023-12-10T19:18:17.894 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:18:17.894 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_43 2023-12-10T19:18:17.894 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:18:17.894 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:18:17.894 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:17.895 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_43 2023-12-10T19:18:17.895 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = journal ] 2023-12-10T19:18:17.895 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = snapshot ] 2023-12-10T19:18:17.895 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_43 2023-12-10T19:18:17.895 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:18:17.895 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:18:17.895 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:17.895 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_43 2023-12-10T19:18:17.895 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster2-mirror-image_43.status 2023-12-10T19:18:17.895 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2-mirror-image_43.status 2023-12-10T19:18:17.895 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:18:17.896 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_43.status 2023-12-10T19:18:17.896 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-image_43.status 2023-12-10T19:18:17.896 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-image_43.status 2023-12-10T19:18:17.896 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:18:17.898 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_43.status 2023-12-10T19:18:17.898 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mirror_image_snapshot cluster2 mirror image_43 2023-12-10T19:18:17.898 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:18:17.898 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:17.898 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_43 2023-12-10T19:18:17.898 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_43 2023-12-10T19:18:18.471 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Snapshot ID: 203 2023-12-10T19:18:18.488 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_43 /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_43.status 2023-12-10T19:18:18.488 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:18:18.488 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:18.488 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_43 2023-12-10T19:18:18.488 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_43.status 2023-12-10T19:18:18.488 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_43 --format xml 2023-12-10T19:18:18.489 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:18:18.542 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-10T19:18:18.544 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snapshot_id=203 2023-12-10T19:18:18.544 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:18:18.544 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.2 2023-12-10T19:18:18.746 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_43.status 2023-12-10T19:18:18.747 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:18.747 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:18.747 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_43 2023-12-10T19:18:18.747 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_43.status 2023-12-10T19:18:18.747 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2023-12-10T19:18:18.747 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:18:18.814 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:18:18.817 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=110 2023-12-10T19:18:18.817 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 203 = 110 2023-12-10T19:18:18.817 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.4 2023-12-10T19:18:19.219 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_43.status 2023-12-10T19:18:19.219 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:19.219 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:19.219 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_43 2023-12-10T19:18:19.219 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_43.status 2023-12-10T19:18:19.219 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2023-12-10T19:18:19.219 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:18:19.286 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:18:19.287 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:18:19.288 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:18:19.288 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:18:19.288 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:18:19.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:18:19.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 203 = 2023-12-10T19:18:19.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.8 2023-12-10T19:18:20.091 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_43.status 2023-12-10T19:18:20.091 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:20.091 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:20.091 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_43 2023-12-10T19:18:20.091 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_43.status 2023-12-10T19:18:20.091 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2023-12-10T19:18:20.091 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:18:20.176 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:18:20.176 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:18:20.178 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:18:20.178 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:18:20.179 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:18:20.179 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:18:20.179 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 203 = 2023-12-10T19:18:20.179 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1.6 2023-12-10T19:18:21.780 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_43.status 2023-12-10T19:18:21.780 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:21.780 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:21.780 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_43 2023-12-10T19:18:21.780 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_43.status 2023-12-10T19:18:21.780 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2023-12-10T19:18:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:18:21.849 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:18:21.852 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=203 2023-12-10T19:18:21.852 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 203 = 203 2023-12-10T19:18:21.852 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:18:21.852 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snap_present cluster1 mirror image_43 snap 2023-12-10T19:18:21.852 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:21.852 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:21.852 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_43 2023-12-10T19:18:21.852 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:18:21.852 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:18:21.852 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:18:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 info mirror/image_43@snap 2023-12-10T19:18:22.927 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_43': 2023-12-10T19:18:22.927 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:18:22.927 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:18:22.927 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 2 2023-12-10T19:18:22.927 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 128b2b092a8c 2023-12-10T19:18:22.927 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.128b2b092a8c 2023-12-10T19:18:22.928 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:18:22.928 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering, non-primary 2023-12-10T19:18:22.928 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:18:22.928 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:18:22.928 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:10:07 2023 2023-12-10T19:18:22.928 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:10:07 2023 2023-12-10T19:18:22.928 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:10:07 2023 2023-12-10T19:18:22.928 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: protected: False 2023-12-10T19:18:22.928 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:18:22.928 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:18:22.928 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: bc83a314-8e55-4363-aaa1-1ab49240de8f 2023-12-10T19:18:22.928 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: false 2023-12-10T19:18:22.932 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:18:22.933 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror image_43 snap 2023-12-10T19:18:22.933 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:22.933 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_43 2023-12-10T19:18:22.933 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:18:22.933 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:18:22.933 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_43.export 2023-12-10T19:18:22.933 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_43.export 2023-12-10T19:18:22.933 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_43.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_43.export 2023-12-10T19:18:22.933 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export image_43@snap /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_43.export 2023-12-10T19:18:23.134 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:18:23.143 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export image_43@snap /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_43.export 2023-12-10T19:18:23.384 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:18:23.394 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_43.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_43.export 2023-12-10T19:18:23.470 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_43.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_43.export 2023-12-10T19:18:23.509 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:18:23.509 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_44 2023-12-10T19:18:23.509 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror image_44 snap 2023-12-10T19:18:23.509 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:18:23.509 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:23.509 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_44 2023-12-10T19:18:23.509 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:18:23.509 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create image_44@snap --debug-rbd=20 --debug-journaler=20 2023-12-10T19:18:23.785 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_started cluster1 mirror image_44 2023-12-10T19:18:23.785 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:23.785 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:23.785 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_44 2023-12-10T19:18:23.785 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_state cluster1 mirror image_44 started 2023-12-10T19:18:23.785 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:23.785 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:23.785 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_44 2023-12-10T19:18:23.785 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=started 2023-12-10T19:18:23.785 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:18:23.785 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:18:24.786 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_replay_state cluster1 mirror image_44 started 2023-12-10T19:18:24.786 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:24.786 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:24.786 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_44 2023-12-10T19:18:24.786 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=started 2023-12-10T19:18:24.786 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_result 2023-12-10T19:18:24.786 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=stopped 2023-12-10T19:18:24.787 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_44 2023-12-10T19:18:24.787 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ + local cluster_instance=cluster1 2023-12-10T19:18:24.787 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:18:24.787 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:grep -i state 2023-12-10T19:18:24.787 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:24.787 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance=cluster1 2023-12-10T19:18:24.787 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loop_instance 2023-12-10T19:18:24.787 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:18:24.788 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T19:18:24.788 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T19:18:24.790 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_44 2023-12-10T19:18:24.790 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T19:18:24.790 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:18:24.790 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:18:24.790 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T19:18:24.790 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:18:24.790 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:18:24.790 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:18:24.790 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:18:24.790 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:18:24.790 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:18:24.790 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:18:24.791 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:18:24.791 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:18:24.791 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:18:24.791 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T19:18:24.791 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T19:18:24.791 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:24.791 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:18:24.791 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T19:18:24.791 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:18:24.791 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:18:24.791 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:18:24.791 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:18:24.791 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:18:24.791 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:18:24.791 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:18:24.792 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:18:24.792 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:18:24.792 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:18:24.792 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T19:18:24.815 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:18:24.815 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:18:24.815 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:18:24.815 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_44 2023-12-10T19:18:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2023-12-10T19:18:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:18:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:18:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_44 2023-12-10T19:18:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:1 2023-12-10T19:18:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:18:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:18:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-10T19:18:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:18:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:18:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:18:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:18:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:18:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:18:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:18:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:18:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:18:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:18:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-10T19:18:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:1 2023-12-10T19:18:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:18:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-10T19:18:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:18:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:18:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:18:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:18:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:18:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:18:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:18:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:18:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:18:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:18:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-10T19:18:24.952 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:18:24.952 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:18:24.952 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:18:24.952 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_44 2023-12-10T19:18:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2023-12-10T19:18:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:18:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:18:25.069 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_44 2023-12-10T19:18:25.069 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:2 2023-12-10T19:18:25.069 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:18:25.069 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:18:25.069 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-10T19:18:25.069 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:18:25.070 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:18:25.070 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:18:25.070 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:18:25.070 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:18:25.070 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:18:25.070 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:18:25.070 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:18:25.070 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:18:25.070 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:18:25.070 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-10T19:18:25.070 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:2 2023-12-10T19:18:25.070 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:25.070 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:18:25.070 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-10T19:18:25.070 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:18:25.071 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:18:25.071 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:18:25.071 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:18:25.071 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:18:25.071 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:18:25.071 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:18:25.071 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:18:25.071 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:18:25.071 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:18:25.071 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-10T19:18:25.093 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:18:25.093 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:18:25.093 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:18:25.093 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_44 2023-12-10T19:18:25.219 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2023-12-10T19:18:25.220 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:18:25.220 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:18:25.221 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_44 2023-12-10T19:18:25.221 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:3 2023-12-10T19:18:25.221 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:18:25.221 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:18:25.221 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-10T19:18:25.221 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:3 2023-12-10T19:18:25.221 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:18:25.221 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:18:25.222 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:18:25.222 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:18:25.222 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 3 = cluster1:3 ] 2023-12-10T19:18:25.222 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:18:25.222 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:18:25.222 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=3 2023-12-10T19:18:25.222 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:18:25.225 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-10T19:18:25.226 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:3 2023-12-10T19:18:25.226 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:25.226 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:18:25.226 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-10T19:18:25.226 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:3 2023-12-10T19:18:25.226 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:18:25.226 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:18:25.226 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:18:25.226 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:18:25.226 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 3 = cluster1:3 ] 2023-12-10T19:18:25.226 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:18:25.226 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:18:25.226 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=3 2023-12-10T19:18:25.226 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:18:25.227 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-10T19:18:25.247 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T19:18:25.247 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T19:18:25.247 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T19:18:25.247 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_44 2023-12-10T19:18:25.382 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:18:25.383 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ status_result= "state": "Replaying" 2023-12-10T19:18:25.383 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo "state": "Replaying" 2023-12-10T19:18:25.383 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep -i Replaying 2023-12-10T19:18:25.384 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "state": "Replaying" 2023-12-10T19:18:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=started 2023-12-10T19:18:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test started = started 2023-12-10T19:18:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:18:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_44 2023-12-10T19:18:25.386 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:18:25.386 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:18:25.386 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:25.386 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_44 2023-12-10T19:18:25.386 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = journal ] 2023-12-10T19:18:25.386 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = snapshot ] 2023-12-10T19:18:25.386 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_44 2023-12-10T19:18:25.386 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:18:25.386 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:18:25.386 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:25.386 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_44 2023-12-10T19:18:25.386 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster2-mirror-image_44.status 2023-12-10T19:18:25.387 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2-mirror-image_44.status 2023-12-10T19:18:25.387 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:18:25.387 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_44.status 2023-12-10T19:18:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-image_44.status 2023-12-10T19:18:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-image_44.status 2023-12-10T19:18:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:18:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_44.status 2023-12-10T19:18:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mirror_image_snapshot cluster2 mirror image_44 2023-12-10T19:18:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:18:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_44 2023-12-10T19:18:25.390 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_44 2023-12-10T19:18:26.455 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Snapshot ID: 205 2023-12-10T19:18:26.466 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_44 /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_44.status 2023-12-10T19:18:26.467 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:18:26.467 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:26.467 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_44 2023-12-10T19:18:26.467 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_44.status 2023-12-10T19:18:26.467 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_44 --format xml 2023-12-10T19:18:26.467 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:18:26.523 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-10T19:18:26.525 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snapshot_id=205 2023-12-10T19:18:26.526 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:18:26.526 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.2 2023-12-10T19:18:26.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_44.status 2023-12-10T19:18:26.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:26.728 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:26.728 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_44 2023-12-10T19:18:26.728 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_44.status 2023-12-10T19:18:26.728 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2023-12-10T19:18:26.728 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:18:26.795 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:18:26.799 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=111 2023-12-10T19:18:26.799 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 205 = 111 2023-12-10T19:18:26.799 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.4 2023-12-10T19:18:27.201 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_44.status 2023-12-10T19:18:27.201 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:27.201 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:27.201 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_44 2023-12-10T19:18:27.201 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_44.status 2023-12-10T19:18:27.202 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2023-12-10T19:18:27.202 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:18:27.272 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:18:27.273 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:18:27.274 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:18:27.274 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:18:27.275 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:18:27.275 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:18:27.275 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 205 = 2023-12-10T19:18:27.275 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.8 2023-12-10T19:18:28.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_44.status 2023-12-10T19:18:28.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:28.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:28.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_44 2023-12-10T19:18:28.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_44.status 2023-12-10T19:18:28.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2023-12-10T19:18:28.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:18:28.144 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:18:28.147 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=205 2023-12-10T19:18:28.147 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 205 = 205 2023-12-10T19:18:28.147 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:18:28.147 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snap_present cluster1 mirror image_44 snap 2023-12-10T19:18:28.147 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:28.147 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:28.147 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_44 2023-12-10T19:18:28.147 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:18:28.147 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:18:28.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:18:29.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 info mirror/image_44@snap 2023-12-10T19:18:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_44': 2023-12-10T19:18:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:18:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:18:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 2 2023-12-10T19:18:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 12a8b2ea908b 2023-12-10T19:18:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.12a8b2ea908b 2023-12-10T19:18:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:18:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering, non-primary 2023-12-10T19:18:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:18:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:18:29.214 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:10:15 2023 2023-12-10T19:18:29.214 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:10:15 2023 2023-12-10T19:18:29.214 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:10:15 2023 2023-12-10T19:18:29.214 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: protected: False 2023-12-10T19:18:29.214 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:18:29.214 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:18:29.214 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: e95c6ade-d549-422b-a9cd-187a5dd6a0d6 2023-12-10T19:18:29.214 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: false 2023-12-10T19:18:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:18:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror image_44 snap 2023-12-10T19:18:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_44 2023-12-10T19:18:29.218 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:18:29.218 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:18:29.218 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_44.export 2023-12-10T19:18:29.218 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_44.export 2023-12-10T19:18:29.218 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_44.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_44.export 2023-12-10T19:18:29.218 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export image_44@snap /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_44.export 2023-12-10T19:18:29.408 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:18:29.418 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export image_44@snap /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_44.export 2023-12-10T19:18:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:18:29.783 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_44.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_44.export 2023-12-10T19:18:29.858 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_44.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_44.export 2023-12-10T19:18:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:18:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_45 2023-12-10T19:18:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror image_45 snap 2023-12-10T19:18:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:18:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_45 2023-12-10T19:18:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:18:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create image_45@snap --debug-rbd=20 --debug-journaler=20 2023-12-10T19:18:30.781 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_started cluster1 mirror image_45 2023-12-10T19:18:30.781 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:30.781 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:30.781 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_45 2023-12-10T19:18:30.781 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_state cluster1 mirror image_45 started 2023-12-10T19:18:30.781 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:30.781 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:30.781 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_45 2023-12-10T19:18:30.781 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=started 2023-12-10T19:18:30.781 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:18:30.781 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:18:31.781 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_replay_state cluster1 mirror image_45 started 2023-12-10T19:18:31.782 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:31.782 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:31.782 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_45 2023-12-10T19:18:31.782 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=started 2023-12-10T19:18:31.782 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_result 2023-12-10T19:18:31.782 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=stopped 2023-12-10T19:18:31.783 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ + admin_daemons cluster1 rbd mirror status mirror/image_45grep 2023-12-10T19:18:31.783 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: -i state 2023-12-10T19:18:31.783 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_instance=cluster1 2023-12-10T19:18:31.783 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:18:31.783 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:31.783 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance=cluster1 2023-12-10T19:18:31.783 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loop_instance 2023-12-10T19:18:31.783 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:18:31.784 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T19:18:31.784 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T19:18:31.785 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_45 2023-12-10T19:18:31.785 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T19:18:31.786 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:18:31.786 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:18:31.786 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T19:18:31.786 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:18:31.786 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:18:31.786 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:18:31.786 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:18:31.786 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:18:31.786 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:18:31.786 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:18:31.786 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:18:31.786 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:18:31.786 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:18:31.786 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T19:18:31.786 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T19:18:31.787 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:31.787 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:18:31.787 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T19:18:31.787 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:18:31.787 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:18:31.787 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:18:31.787 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:18:31.787 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:18:31.787 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:18:31.787 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:18:31.787 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:18:31.787 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:18:31.787 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:18:31.787 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T19:18:31.810 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:18:31.811 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:18:31.811 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:18:31.811 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_45 2023-12-10T19:18:31.922 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2023-12-10T19:18:31.922 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:18:31.922 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:18:31.925 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_45 2023-12-10T19:18:31.925 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:1 2023-12-10T19:18:31.925 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:18:31.925 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:18:31.926 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-10T19:18:31.926 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:18:31.926 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:18:31.926 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:18:31.926 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:18:31.926 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:18:31.926 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:18:31.926 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:18:31.926 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:18:31.926 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:18:31.926 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:18:31.926 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-10T19:18:31.926 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:1 2023-12-10T19:18:31.926 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:31.926 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:18:31.927 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-10T19:18:31.927 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:18:31.927 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:18:31.927 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:18:31.927 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:18:31.927 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:18:31.927 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:18:31.927 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:18:31.927 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:18:31.927 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:18:31.927 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:18:31.927 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-10T19:18:31.948 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:18:31.948 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:18:31.948 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:18:31.948 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_45 2023-12-10T19:18:32.065 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2023-12-10T19:18:32.065 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:18:32.065 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:18:32.067 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_45 2023-12-10T19:18:32.067 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:2 2023-12-10T19:18:32.067 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:18:32.067 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:18:32.067 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-10T19:18:32.067 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:18:32.067 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:18:32.068 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:18:32.068 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:18:32.068 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:18:32.068 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:18:32.068 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:18:32.068 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:18:32.068 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:18:32.068 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:18:32.068 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-10T19:18:32.068 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:2 2023-12-10T19:18:32.068 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:32.068 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:18:32.068 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-10T19:18:32.068 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:18:32.068 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:18:32.068 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:18:32.069 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:18:32.069 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:18:32.069 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:18:32.069 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:18:32.069 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:18:32.069 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:18:32.069 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:18:32.069 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-10T19:18:32.092 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:18:32.092 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:18:32.092 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:18:32.092 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_45 2023-12-10T19:18:32.214 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:18:32.215 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ status_result= "state": "Replaying" 2023-12-10T19:18:32.215 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo "state": "Replaying" 2023-12-10T19:18:32.215 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep -i Replaying 2023-12-10T19:18:32.216 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "state": "Replaying" 2023-12-10T19:18:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=started 2023-12-10T19:18:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test started = started 2023-12-10T19:18:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:18:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_45 2023-12-10T19:18:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:18:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:18:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_45 2023-12-10T19:18:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = journal ] 2023-12-10T19:18:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = snapshot ] 2023-12-10T19:18:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_45 2023-12-10T19:18:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:18:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:18:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_45 2023-12-10T19:18:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster2-mirror-image_45.status 2023-12-10T19:18:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2-mirror-image_45.status 2023-12-10T19:18:32.219 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:18:32.219 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_45.status 2023-12-10T19:18:32.221 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-image_45.status 2023-12-10T19:18:32.221 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-image_45.status 2023-12-10T19:18:32.221 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:18:32.222 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_45.status 2023-12-10T19:18:32.222 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mirror_image_snapshot cluster2 mirror image_45 2023-12-10T19:18:32.222 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:18:32.222 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:32.222 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_45 2023-12-10T19:18:32.222 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_45 2023-12-10T19:18:32.805 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Snapshot ID: 207 2023-12-10T19:18:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_45 /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_45.status 2023-12-10T19:18:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:18:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_45 2023-12-10T19:18:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_45.status 2023-12-10T19:18:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_45 --format xml 2023-12-10T19:18:32.819 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:18:32.872 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-10T19:18:32.874 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snapshot_id=207 2023-12-10T19:18:32.875 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:18:32.875 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.2 2023-12-10T19:18:33.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_45.status 2023-12-10T19:18:33.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:33.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:33.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_45 2023-12-10T19:18:33.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_45.status 2023-12-10T19:18:33.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2023-12-10T19:18:33.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:18:33.146 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:18:33.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=112 2023-12-10T19:18:33.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 207 = 112 2023-12-10T19:18:33.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.4 2023-12-10T19:18:33.550 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_45.status 2023-12-10T19:18:33.550 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:33.550 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:33.550 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_45 2023-12-10T19:18:33.550 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_45.status 2023-12-10T19:18:33.550 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2023-12-10T19:18:33.551 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:18:33.621 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:18:33.623 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=112 2023-12-10T19:18:33.623 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 207 = 112 2023-12-10T19:18:33.624 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.8 2023-12-10T19:18:33.880 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5164659770995859) > weight (0.25) 2023-12-10T19:18:33.880 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.7598103708453388) > weight (0.25) 2023-12-10T19:18:33.880 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.8485176910741662) > weight (0.25) 2023-12-10T19:18:33.880 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.7890176491482784) > weight (0.25) 2023-12-10T19:18:33.881 INFO:tasks.rbd_mirror_thrash:waiting for 118 secs before thrashing 2023-12-10T19:18:34.425 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_45.status 2023-12-10T19:18:34.425 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:34.425 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:34.425 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_45 2023-12-10T19:18:34.425 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_45.status 2023-12-10T19:18:34.426 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2023-12-10T19:18:34.426 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:18:34.723 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:18:34.724 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:18:34.728 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:18:34.728 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:18:34.728 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:18:34.729 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:18:34.729 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 207 = 2023-12-10T19:18:34.729 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1.6 2023-12-10T19:18:36.330 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_45.status 2023-12-10T19:18:36.331 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:36.331 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:36.331 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_45 2023-12-10T19:18:36.331 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_45.status 2023-12-10T19:18:36.331 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2023-12-10T19:18:36.331 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:18:36.402 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:18:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=207 2023-12-10T19:18:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 207 = 207 2023-12-10T19:18:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:18:36.406 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snap_present cluster1 mirror image_45 snap 2023-12-10T19:18:36.406 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:36.406 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:36.406 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_45 2023-12-10T19:18:36.406 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:18:36.406 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:18:36.406 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:18:37.406 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 info mirror/image_45@snap 2023-12-10T19:18:37.472 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_45': 2023-12-10T19:18:37.472 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:18:37.472 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:18:37.472 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 2 2023-12-10T19:18:37.472 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 10c03e893e0d 2023-12-10T19:18:37.472 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.10c03e893e0d 2023-12-10T19:18:37.472 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:18:37.472 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering, non-primary 2023-12-10T19:18:37.472 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:18:37.472 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:18:37.472 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:10:22 2023 2023-12-10T19:18:37.472 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:10:22 2023 2023-12-10T19:18:37.473 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:10:22 2023 2023-12-10T19:18:37.473 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: protected: False 2023-12-10T19:18:37.473 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:18:37.473 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:18:37.473 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: 40a91520-4e7a-4ba5-aaf0-64312a2dcb5a 2023-12-10T19:18:37.473 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: false 2023-12-10T19:18:37.476 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:18:37.477 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror image_45 snap 2023-12-10T19:18:37.477 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:37.477 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_45 2023-12-10T19:18:37.477 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:18:37.477 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:18:37.477 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_45.export 2023-12-10T19:18:37.477 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_45.export 2023-12-10T19:18:37.477 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_45.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_45.export 2023-12-10T19:18:37.477 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export image_45@snap /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_45.export 2023-12-10T19:18:37.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:18:37.684 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export image_45@snap /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_45.export 2023-12-10T19:18:38.108 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:18:38.126 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_45.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_45.export 2023-12-10T19:18:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_45.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_45.export 2023-12-10T19:18:38.246 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:18:38.247 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_46 2023-12-10T19:18:38.247 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror image_46 snap 2023-12-10T19:18:38.247 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:18:38.247 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:38.247 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_46 2023-12-10T19:18:38.247 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:18:38.247 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create image_46@snap --debug-rbd=20 --debug-journaler=20 2023-12-10T19:18:38.789 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_started cluster1 mirror image_46 2023-12-10T19:18:38.789 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:38.789 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:38.789 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_46 2023-12-10T19:18:38.789 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_state cluster1 mirror image_46 started 2023-12-10T19:18:38.790 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:38.790 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:38.790 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_46 2023-12-10T19:18:38.790 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=started 2023-12-10T19:18:38.790 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:18:38.790 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:18:39.791 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_replay_state cluster1 mirror image_46 started 2023-12-10T19:18:39.791 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:39.792 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:39.792 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_46 2023-12-10T19:18:39.792 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=started 2023-12-10T19:18:39.792 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_result 2023-12-10T19:18:39.792 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=stopped 2023-12-10T19:18:39.792 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemons cluster1+ rbd mirror status mirror/image_46 2023-12-10T19:18:39.793 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:grep -i state 2023-12-10T19:18:39.793 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_instance=cluster1 2023-12-10T19:18:39.793 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:18:39.793 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:39.793 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance=cluster1 2023-12-10T19:18:39.793 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loop_instance 2023-12-10T19:18:39.794 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:18:39.798 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T19:18:39.798 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T19:18:39.798 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_46 2023-12-10T19:18:39.799 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T19:18:39.799 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:18:39.799 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:18:39.799 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T19:18:39.800 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:18:39.800 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:18:39.800 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:18:39.800 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:18:39.801 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:18:39.801 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:18:39.801 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:18:39.801 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:18:39.801 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:18:39.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:18:39.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T19:18:39.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T19:18:39.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:39.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:18:39.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T19:18:39.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:18:39.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:18:39.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:18:39.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:18:39.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:18:39.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:18:39.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:18:39.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:18:39.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:18:39.803 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:18:39.803 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T19:18:39.822 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:18:39.822 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:18:39.822 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:18:39.822 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_46 2023-12-10T19:18:39.937 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2023-12-10T19:18:39.937 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:18:39.937 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:18:39.940 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_46 2023-12-10T19:18:39.940 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:1 2023-12-10T19:18:39.940 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:18:39.940 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:18:39.940 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-10T19:18:39.940 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:18:39.940 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:18:39.940 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:18:39.940 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:18:39.940 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:18:39.940 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:18:39.940 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:18:39.940 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:18:39.940 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:18:39.940 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:18:39.941 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-10T19:18:39.941 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:1 2023-12-10T19:18:39.941 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:39.941 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:18:39.941 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-10T19:18:39.941 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:18:39.941 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:18:39.941 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:18:39.941 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:18:39.941 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:18:39.941 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:18:39.941 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:18:39.941 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:18:39.941 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:18:39.942 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:18:39.942 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-10T19:18:39.964 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:18:39.964 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:18:39.964 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:18:39.964 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_46 2023-12-10T19:18:40.072 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2023-12-10T19:18:40.072 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:18:40.072 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:18:40.075 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_46 2023-12-10T19:18:40.075 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:2 2023-12-10T19:18:40.075 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:18:40.075 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:18:40.075 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-10T19:18:40.075 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:18:40.075 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:18:40.075 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:18:40.075 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:18:40.075 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:18:40.075 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:18:40.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:18:40.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:18:40.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:18:40.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:18:40.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-10T19:18:40.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:2 2023-12-10T19:18:40.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:40.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:18:40.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-10T19:18:40.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:18:40.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:18:40.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:18:40.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:18:40.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:18:40.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:18:40.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:18:40.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:18:40.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:18:40.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:18:40.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-10T19:18:40.099 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:18:40.099 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:18:40.099 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:18:40.099 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_46 2023-12-10T19:18:40.225 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:18:40.226 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ status_result= "state": "Replaying" 2023-12-10T19:18:40.226 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo "state": "Replaying" 2023-12-10T19:18:40.226 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep -i Replaying 2023-12-10T19:18:40.227 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "state": "Replaying" 2023-12-10T19:18:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=started 2023-12-10T19:18:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test started = started 2023-12-10T19:18:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:18:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_46 2023-12-10T19:18:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:18:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:18:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_46 2023-12-10T19:18:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = journal ] 2023-12-10T19:18:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = snapshot ] 2023-12-10T19:18:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_46 2023-12-10T19:18:40.231 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:18:40.231 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:18:40.231 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:40.231 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_46 2023-12-10T19:18:40.231 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster2-mirror-image_46.status 2023-12-10T19:18:40.231 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ + echo cluster2-mirror-image_46.status 2023-12-10T19:18:40.231 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:sed -e s|[/ ]|_|g 2023-12-10T19:18:40.231 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_46.status 2023-12-10T19:18:40.231 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-image_46.status 2023-12-10T19:18:40.231 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-image_46.status 2023-12-10T19:18:40.231 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:18:40.232 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_46.status 2023-12-10T19:18:40.232 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mirror_image_snapshot cluster2 mirror image_46 2023-12-10T19:18:40.232 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:18:40.232 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:40.232 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_46 2023-12-10T19:18:40.232 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_46 2023-12-10T19:18:40.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Snapshot ID: 209 2023-12-10T19:18:40.815 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_46 /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_46.status 2023-12-10T19:18:40.815 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:18:40.816 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:40.816 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_46 2023-12-10T19:18:40.816 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_46.status 2023-12-10T19:18:40.816 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_46 --format xml 2023-12-10T19:18:40.816 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:18:40.871 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-10T19:18:40.874 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snapshot_id=209 2023-12-10T19:18:40.874 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:18:40.874 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.2 2023-12-10T19:18:41.075 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_46.status 2023-12-10T19:18:41.075 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:41.075 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:41.075 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_46 2023-12-10T19:18:41.075 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_46.status 2023-12-10T19:18:41.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2023-12-10T19:18:41.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:18:41.144 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:18:41.147 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=113 2023-12-10T19:18:41.147 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 209 = 113 2023-12-10T19:18:41.147 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.4 2023-12-10T19:18:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_46.status 2023-12-10T19:18:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_46 2023-12-10T19:18:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_46.status 2023-12-10T19:18:41.549 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2023-12-10T19:18:41.549 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:18:41.618 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:18:41.621 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=113 2023-12-10T19:18:41.621 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 209 = 113 2023-12-10T19:18:41.621 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.8 2023-12-10T19:18:42.422 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_46.status 2023-12-10T19:18:42.422 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:42.422 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:42.422 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_46 2023-12-10T19:18:42.422 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_46.status 2023-12-10T19:18:42.422 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2023-12-10T19:18:42.423 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:18:42.497 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:18:42.497 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:18:42.499 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:18:42.499 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:18:42.499 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:18:42.500 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:18:42.500 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 209 = 2023-12-10T19:18:42.500 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1.6 2023-12-10T19:18:44.101 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_46.status 2023-12-10T19:18:44.102 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:44.102 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:44.102 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_46 2023-12-10T19:18:44.102 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_46.status 2023-12-10T19:18:44.102 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2023-12-10T19:18:44.102 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:18:44.168 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:18:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=209 2023-12-10T19:18:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 209 = 209 2023-12-10T19:18:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:18:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snap_present cluster1 mirror image_46 snap 2023-12-10T19:18:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_46 2023-12-10T19:18:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:18:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:18:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:18:45.172 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 info mirror/image_46@snap 2023-12-10T19:18:45.236 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_46': 2023-12-10T19:18:45.236 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:18:45.236 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:18:45.236 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 2 2023-12-10T19:18:45.236 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 10e8fcdb213d 2023-12-10T19:18:45.236 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.10e8fcdb213d 2023-12-10T19:18:45.236 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:18:45.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering, non-primary 2023-12-10T19:18:45.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:18:45.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:18:45.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:10:34 2023 2023-12-10T19:18:45.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:10:34 2023 2023-12-10T19:18:45.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:10:34 2023 2023-12-10T19:18:45.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: protected: False 2023-12-10T19:18:45.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:18:45.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:18:45.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: 509368f6-035e-4be6-926a-dc4bce662e88 2023-12-10T19:18:45.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: false 2023-12-10T19:18:45.240 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:18:45.240 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror image_46 snap 2023-12-10T19:18:45.240 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:45.240 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_46 2023-12-10T19:18:45.240 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:18:45.240 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:18:45.240 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_46.export 2023-12-10T19:18:45.240 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_46.export 2023-12-10T19:18:45.241 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_46.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_46.export 2023-12-10T19:18:45.241 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export image_46@snap /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_46.export 2023-12-10T19:18:45.458 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:18:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export image_46@snap /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_46.export 2023-12-10T19:18:45.801 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:18:45.813 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_46.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_46.export 2023-12-10T19:18:45.890 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_46.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_46.export 2023-12-10T19:18:45.931 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:18:45.931 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_47 2023-12-10T19:18:45.932 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror image_47 snap 2023-12-10T19:18:45.932 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:18:45.932 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:45.932 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_47 2023-12-10T19:18:45.932 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:18:45.932 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create image_47@snap --debug-rbd=20 --debug-journaler=20 2023-12-10T19:18:46.453 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_started cluster1 mirror image_47 2023-12-10T19:18:46.453 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:46.453 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:46.453 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_47 2023-12-10T19:18:46.453 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_state cluster1 mirror image_47 started 2023-12-10T19:18:46.453 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:46.453 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:46.453 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_47 2023-12-10T19:18:46.453 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=started 2023-12-10T19:18:46.453 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:18:46.453 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:18:47.454 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_replay_state cluster1 mirror image_47 started 2023-12-10T19:18:47.454 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:47.454 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:47.454 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_47 2023-12-10T19:18:47.454 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=started 2023-12-10T19:18:47.454 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_result 2023-12-10T19:18:47.454 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=stopped 2023-12-10T19:18:47.455 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemons cluster1+ rbd mirror status mirror/image_47 2023-12-10T19:18:47.455 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:grep -i state 2023-12-10T19:18:47.455 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_instance=cluster1 2023-12-10T19:18:47.455 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:18:47.455 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:47.455 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance=cluster1 2023-12-10T19:18:47.455 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loop_instance 2023-12-10T19:18:47.455 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:18:47.456 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T19:18:47.456 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T19:18:47.457 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_47 2023-12-10T19:18:47.457 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T19:18:47.458 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:18:47.458 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:18:47.458 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T19:18:47.458 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:18:47.458 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:18:47.458 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:18:47.458 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:18:47.458 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:18:47.458 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:18:47.458 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:18:47.458 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:18:47.458 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:18:47.458 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:18:47.458 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T19:18:47.459 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T19:18:47.459 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:47.459 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:18:47.459 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T19:18:47.459 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:18:47.459 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:18:47.459 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:18:47.459 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:18:47.459 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:18:47.459 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:18:47.459 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:18:47.459 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:18:47.459 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:18:47.459 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:18:47.460 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T19:18:47.480 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:18:47.480 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:18:47.480 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:18:47.480 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_47 2023-12-10T19:18:47.604 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:18:47.605 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ status_result= "state": "Replaying" 2023-12-10T19:18:47.605 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo "state": "Replaying" 2023-12-10T19:18:47.605 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep -i Replaying 2023-12-10T19:18:47.606 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "state": "Replaying" 2023-12-10T19:18:47.607 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=started 2023-12-10T19:18:47.607 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test started = started 2023-12-10T19:18:47.607 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:18:47.607 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_47 2023-12-10T19:18:47.607 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:18:47.607 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:18:47.608 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:47.608 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_47 2023-12-10T19:18:47.608 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = journal ] 2023-12-10T19:18:47.608 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = snapshot ] 2023-12-10T19:18:47.608 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_47 2023-12-10T19:18:47.608 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:18:47.608 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:18:47.608 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:47.608 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_47 2023-12-10T19:18:47.608 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster2-mirror-image_47.status 2023-12-10T19:18:47.608 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2-mirror-image_47.status 2023-12-10T19:18:47.608 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:18:47.609 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_47.status 2023-12-10T19:18:47.609 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-image_47.status 2023-12-10T19:18:47.609 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-image_47.status 2023-12-10T19:18:47.609 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:18:47.611 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_47.status 2023-12-10T19:18:47.611 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mirror_image_snapshot cluster2 mirror image_47 2023-12-10T19:18:47.611 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:18:47.611 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:47.611 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_47 2023-12-10T19:18:47.611 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_47 2023-12-10T19:18:48.484 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Snapshot ID: 211 2023-12-10T19:18:48.496 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_47 /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_47.status 2023-12-10T19:18:48.496 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:18:48.496 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:48.497 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_47 2023-12-10T19:18:48.497 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_47.status 2023-12-10T19:18:48.497 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_47 --format xml 2023-12-10T19:18:48.497 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:18:48.553 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-10T19:18:48.555 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snapshot_id=211 2023-12-10T19:18:48.555 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:18:48.555 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.2 2023-12-10T19:18:48.757 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_47.status 2023-12-10T19:18:48.757 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:48.757 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:48.757 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_47 2023-12-10T19:18:48.757 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_47.status 2023-12-10T19:18:48.757 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2023-12-10T19:18:48.757 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:18:48.825 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:18:48.825 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:18:48.827 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:18:48.827 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:18:48.827 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:18:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:18:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 211 = 2023-12-10T19:18:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.4 2023-12-10T19:18:49.229 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_47.status 2023-12-10T19:18:49.229 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:49.229 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:49.229 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_47 2023-12-10T19:18:49.229 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_47.status 2023-12-10T19:18:49.229 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2023-12-10T19:18:49.229 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:18:49.297 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:18:49.298 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:18:49.299 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:18:49.300 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:18:49.300 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:18:49.300 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:18:49.300 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 211 = 2023-12-10T19:18:49.300 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.8 2023-12-10T19:18:50.101 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_47.status 2023-12-10T19:18:50.101 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:50.101 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:50.101 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_47 2023-12-10T19:18:50.101 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_47.status 2023-12-10T19:18:50.101 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2023-12-10T19:18:50.101 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:18:50.161 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:18:50.163 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=211 2023-12-10T19:18:50.163 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 211 = 211 2023-12-10T19:18:50.163 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:18:50.163 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snap_present cluster1 mirror image_47 snap 2023-12-10T19:18:50.163 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:50.163 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:50.163 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_47 2023-12-10T19:18:50.164 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:18:50.164 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:18:50.164 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:18:51.164 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 info mirror/image_47@snap 2023-12-10T19:18:51.227 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_47': 2023-12-10T19:18:51.227 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:18:51.227 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:18:51.227 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 2 2023-12-10T19:18:51.227 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 128b238ded1c 2023-12-10T19:18:51.227 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.128b238ded1c 2023-12-10T19:18:51.227 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:18:51.227 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering, non-primary 2023-12-10T19:18:51.227 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:18:51.227 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:18:51.228 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:10:42 2023 2023-12-10T19:18:51.228 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:10:42 2023 2023-12-10T19:18:51.228 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:10:42 2023 2023-12-10T19:18:51.228 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: protected: False 2023-12-10T19:18:51.228 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:18:51.228 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:18:51.228 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: 35463a96-219e-457b-aaa2-28a56599112e 2023-12-10T19:18:51.228 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: false 2023-12-10T19:18:51.231 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:18:51.231 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror image_47 snap 2023-12-10T19:18:51.231 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:51.231 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_47 2023-12-10T19:18:51.232 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:18:51.232 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:18:51.232 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_47.export 2023-12-10T19:18:51.232 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_47.export 2023-12-10T19:18:51.232 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_47.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_47.export 2023-12-10T19:18:51.232 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export image_47@snap /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_47.export 2023-12-10T19:18:51.437 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:18:51.445 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export image_47@snap /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_47.export 2023-12-10T19:18:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:18:51.754 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_47.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_47.export 2023-12-10T19:18:51.831 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_47.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_47.export 2023-12-10T19:18:51.872 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:18:51.872 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_48 2023-12-10T19:18:51.872 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror image_48 snap 2023-12-10T19:18:51.872 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:18:51.872 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:51.872 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_48 2023-12-10T19:18:51.872 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:18:51.872 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create image_48@snap --debug-rbd=20 --debug-journaler=20 2023-12-10T19:18:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_started cluster1 mirror image_48 2023-12-10T19:18:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_48 2023-12-10T19:18:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_state cluster1 mirror image_48 started 2023-12-10T19:18:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_48 2023-12-10T19:18:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=started 2023-12-10T19:18:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:18:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:18:53.782 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_replay_state cluster1 mirror image_48 started 2023-12-10T19:18:53.783 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:53.783 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:53.783 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_48 2023-12-10T19:18:53.783 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=started 2023-12-10T19:18:53.783 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_result 2023-12-10T19:18:53.783 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=stopped 2023-12-10T19:18:53.784 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_48 2023-12-10T19:18:53.784 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_instance=cluster1 2023-12-10T19:18:53.784 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:18:53.784 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:53.784 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance=cluster1 2023-12-10T19:18:53.784 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loop_instance 2023-12-10T19:18:53.784 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ + sleep 0 2023-12-10T19:18:53.784 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:grep -i state 2023-12-10T19:18:53.785 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T19:18:53.785 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T19:18:53.786 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_48 2023-12-10T19:18:53.787 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T19:18:53.787 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:18:53.787 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:18:53.787 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T19:18:53.787 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:18:53.787 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:18:53.787 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:18:53.787 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:18:53.787 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:18:53.787 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:18:53.787 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:18:53.787 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:18:53.787 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:18:53.787 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:18:53.788 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T19:18:53.788 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T19:18:53.788 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:53.788 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:18:53.788 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T19:18:53.788 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:18:53.788 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:18:53.788 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:18:53.788 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:18:53.788 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:18:53.788 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:18:53.788 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:18:53.788 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:18:53.788 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:18:53.788 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:18:53.788 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T19:18:53.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:18:53.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:18:53.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:18:53.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_48 2023-12-10T19:18:53.873 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7410624137349532) > weight (0.25) 2023-12-10T19:18:53.874 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.6817012039499912) > weight (0.25) 2023-12-10T19:18:53.874 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.3075636570862156) > weight (0.25) 2023-12-10T19:18:53.874 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.6984739634039426) > weight (0.25) 2023-12-10T19:18:53.874 INFO:tasks.rbd_mirror_thrash:waiting for 90 secs before thrashing 2023-12-10T19:18:53.926 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2023-12-10T19:18:53.926 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:18:53.926 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:18:53.928 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_48 2023-12-10T19:18:53.928 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:1 2023-12-10T19:18:53.928 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:18:53.929 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:18:53.929 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-10T19:18:53.929 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:18:53.929 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:18:53.929 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:18:53.929 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:18:53.929 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:18:53.929 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:18:53.929 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:18:53.929 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:18:53.929 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:18:53.929 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:18:53.929 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-10T19:18:53.929 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:1 2023-12-10T19:18:53.929 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:53.929 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:18:53.930 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-10T19:18:53.930 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:18:53.930 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:18:53.930 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:18:53.930 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:18:53.930 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:18:53.930 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:18:53.930 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:18:53.930 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:18:53.930 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:18:53.930 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:18:53.930 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-10T19:18:53.951 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:18:53.951 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:18:53.951 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:18:53.951 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_48 2023-12-10T19:18:54.072 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:18:54.073 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ status_result= "state": "Replaying" 2023-12-10T19:18:54.073 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo "state": "Replaying" 2023-12-10T19:18:54.073 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep -i Replaying 2023-12-10T19:18:54.074 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "state": "Replaying" 2023-12-10T19:18:54.075 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=started 2023-12-10T19:18:54.075 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test started = started 2023-12-10T19:18:54.075 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:18:54.075 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_48 2023-12-10T19:18:54.075 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:18:54.075 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:18:54.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:54.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_48 2023-12-10T19:18:54.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = journal ] 2023-12-10T19:18:54.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = snapshot ] 2023-12-10T19:18:54.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_48 2023-12-10T19:18:54.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:18:54.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:18:54.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:54.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_48 2023-12-10T19:18:54.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster2-mirror-image_48.status 2023-12-10T19:18:54.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2-mirror-image_48.status 2023-12-10T19:18:54.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:18:54.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_48.status 2023-12-10T19:18:54.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-image_48.status 2023-12-10T19:18:54.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-image_48.status 2023-12-10T19:18:54.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:18:54.078 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_48.status 2023-12-10T19:18:54.078 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mirror_image_snapshot cluster2 mirror image_48 2023-12-10T19:18:54.078 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:18:54.078 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:54.078 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_48 2023-12-10T19:18:54.078 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_48 2023-12-10T19:18:54.808 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Snapshot ID: 213 2023-12-10T19:18:54.820 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_48 /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_48.status 2023-12-10T19:18:54.820 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:18:54.820 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:54.821 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_48 2023-12-10T19:18:54.821 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_48.status 2023-12-10T19:18:54.821 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_48 --format xml 2023-12-10T19:18:54.821 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:18:54.875 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-10T19:18:54.878 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snapshot_id=213 2023-12-10T19:18:54.878 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:18:54.878 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.2 2023-12-10T19:18:55.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_48.status 2023-12-10T19:18:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_48 2023-12-10T19:18:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_48.status 2023-12-10T19:18:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2023-12-10T19:18:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:18:55.157 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:18:55.159 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=115 2023-12-10T19:18:55.159 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 213 = 115 2023-12-10T19:18:55.159 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.4 2023-12-10T19:18:55.560 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_48.status 2023-12-10T19:18:55.561 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:55.561 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:55.561 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_48 2023-12-10T19:18:55.561 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_48.status 2023-12-10T19:18:55.561 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2023-12-10T19:18:55.561 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:18:55.628 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:18:55.631 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=115 2023-12-10T19:18:55.631 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 213 = 115 2023-12-10T19:18:55.631 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.8 2023-12-10T19:18:56.432 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_48.status 2023-12-10T19:18:56.432 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:56.432 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:56.433 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_48 2023-12-10T19:18:56.433 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_48.status 2023-12-10T19:18:56.433 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2023-12-10T19:18:56.433 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:18:56.501 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:18:56.501 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:18:56.503 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:18:56.503 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:18:56.503 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:18:56.504 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:18:56.504 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 213 = 2023-12-10T19:18:56.504 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1.6 2023-12-10T19:18:58.105 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_48.status 2023-12-10T19:18:58.105 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:58.105 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:58.105 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_48 2023-12-10T19:18:58.105 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_48.status 2023-12-10T19:18:58.106 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2023-12-10T19:18:58.106 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:18:58.174 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:18:58.177 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=213 2023-12-10T19:18:58.177 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 213 = 213 2023-12-10T19:18:58.177 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:18:58.177 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snap_present cluster1 mirror image_48 snap 2023-12-10T19:18:58.177 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:18:58.177 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:58.177 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_48 2023-12-10T19:18:58.177 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:18:58.177 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:18:58.177 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:18:59.177 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 info mirror/image_48@snap 2023-12-10T19:18:59.240 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_48': 2023-12-10T19:18:59.240 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:18:59.240 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:18:59.240 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 2 2023-12-10T19:18:59.241 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 12a8ad995520 2023-12-10T19:18:59.241 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.12a8ad995520 2023-12-10T19:18:59.241 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:18:59.241 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering, non-primary 2023-12-10T19:18:59.241 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:18:59.241 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:18:59.241 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:10:56 2023 2023-12-10T19:18:59.241 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:10:56 2023 2023-12-10T19:18:59.241 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:10:56 2023 2023-12-10T19:18:59.241 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: protected: False 2023-12-10T19:18:59.241 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:18:59.241 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:18:59.241 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: 935218e9-f0e7-416b-a035-66ab9afc74f5 2023-12-10T19:18:59.241 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: false 2023-12-10T19:18:59.244 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:18:59.244 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror image_48 snap 2023-12-10T19:18:59.244 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:59.244 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_48 2023-12-10T19:18:59.244 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:18:59.244 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:18:59.245 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_48.export 2023-12-10T19:18:59.245 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_48.export 2023-12-10T19:18:59.245 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_48.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_48.export 2023-12-10T19:18:59.245 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export image_48@snap /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_48.export 2023-12-10T19:18:59.432 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:18:59.439 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export image_48@snap /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_48.export 2023-12-10T19:18:59.755 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:18:59.771 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_48.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_48.export 2023-12-10T19:18:59.847 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_48.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_48.export 2023-12-10T19:18:59.888 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:18:59.888 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_49 2023-12-10T19:18:59.888 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror image_49 snap 2023-12-10T19:18:59.888 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:18:59.888 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:18:59.888 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_49 2023-12-10T19:18:59.888 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:18:59.888 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create image_49@snap --debug-rbd=20 --debug-journaler=20 2023-12-10T19:19:00.813 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_started cluster1 mirror image_49 2023-12-10T19:19:00.814 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:19:00.814 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:00.814 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_49 2023-12-10T19:19:00.814 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_state cluster1 mirror image_49 started 2023-12-10T19:19:00.814 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:19:00.814 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:00.814 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_49 2023-12-10T19:19:00.814 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=started 2023-12-10T19:19:00.814 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:19:00.814 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:19:01.815 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_replay_state cluster1 mirror image_49 started 2023-12-10T19:19:01.815 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:19:01.815 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:01.815 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_49 2023-12-10T19:19:01.816 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=started 2023-12-10T19:19:01.816 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_result 2023-12-10T19:19:01.816 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=stopped 2023-12-10T19:19:01.816 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_49 2023-12-10T19:19:01.816 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_instance=cluster1 2023-12-10T19:19:01.816 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:19:01.816 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ + local cluster=cluster1 2023-12-10T19:19:01.816 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance=cluster1 2023-12-10T19:19:01.816 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loop_instance 2023-12-10T19:19:01.816 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:grep -i state+ 2023-12-10T19:19:01.816 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:sleep 0 2023-12-10T19:19:01.819 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T19:19:01.819 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T19:19:01.819 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_49 2023-12-10T19:19:01.819 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T19:19:01.819 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:19:01.819 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:19:01.819 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T19:19:01.819 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:19:01.819 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:19:01.819 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:19:01.819 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:19:01.820 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:19:01.820 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:19:01.820 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:19:01.820 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:19:01.820 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:19:01.820 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:19:01.820 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T19:19:01.820 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T19:19:01.820 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:19:01.820 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:19:01.820 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T19:19:01.820 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:19:01.820 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:19:01.820 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:19:01.820 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:19:01.821 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:19:01.821 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:19:01.821 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:19:01.821 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:19:01.821 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:19:01.821 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:19:01.821 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T19:19:01.842 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:19:01.842 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:19:01.843 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:19:01.843 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_49 2023-12-10T19:19:01.955 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_49'] 2023-12-10T19:19:01.955 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:19:01.955 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:19:01.958 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_49 2023-12-10T19:19:01.958 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:1 2023-12-10T19:19:01.958 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:19:01.958 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:19:01.959 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-10T19:19:01.959 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:19:01.959 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:19:01.959 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:19:01.959 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:19:01.959 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:19:01.959 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:19:01.959 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:19:01.959 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:19:01.959 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:19:01.959 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:19:01.959 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-10T19:19:01.959 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:1 2023-12-10T19:19:01.959 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:19:01.959 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:19:01.959 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-10T19:19:01.960 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:19:01.960 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:19:01.960 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:19:01.960 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:19:01.960 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:19:01.960 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:19:01.960 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:19:01.960 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:19:01.960 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:19:01.960 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:19:01.960 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-10T19:19:01.981 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:19:01.981 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:19:01.981 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:19:01.982 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_49 2023-12-10T19:19:02.091 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_49'] 2023-12-10T19:19:02.091 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:19:02.091 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:19:02.094 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_49 2023-12-10T19:19:02.094 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:2 2023-12-10T19:19:02.094 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:19:02.094 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:19:02.094 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-10T19:19:02.094 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:19:02.094 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:19:02.094 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:19:02.094 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:19:02.094 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:19:02.094 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:19:02.094 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:19:02.095 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:19:02.095 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:19:02.095 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:19:02.095 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-10T19:19:02.095 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:2 2023-12-10T19:19:02.095 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:19:02.095 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:19:02.095 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-10T19:19:02.095 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:19:02.095 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:19:02.095 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:19:02.095 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:19:02.095 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:19:02.095 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:19:02.095 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:19:02.095 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:19:02.095 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:19:02.096 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:19:02.096 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-10T19:19:02.119 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:19:02.119 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:19:02.119 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:19:02.119 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_49 2023-12-10T19:19:02.232 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_49'] 2023-12-10T19:19:02.232 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:19:02.232 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:19:02.234 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_49 2023-12-10T19:19:02.234 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:3 2023-12-10T19:19:02.234 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:19:02.235 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:19:02.235 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-10T19:19:02.235 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:3 2023-12-10T19:19:02.235 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:19:02.235 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:19:02.235 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:19:02.235 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:19:02.235 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 3 = cluster1:3 ] 2023-12-10T19:19:02.235 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:19:02.235 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:19:02.235 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=3 2023-12-10T19:19:02.236 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:19:02.236 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-10T19:19:02.236 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:3 2023-12-10T19:19:02.236 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:19:02.236 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:19:02.236 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-10T19:19:02.236 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:3 2023-12-10T19:19:02.236 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:19:02.236 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:19:02.236 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:19:02.236 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:19:02.236 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 3 = cluster1:3 ] 2023-12-10T19:19:02.236 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:19:02.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:19:02.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=3 2023-12-10T19:19:02.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:19:02.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-10T19:19:02.259 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T19:19:02.259 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T19:19:02.259 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T19:19:02.259 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_49 2023-12-10T19:19:02.386 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:19:02.387 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ status_result= "state": "Replaying" 2023-12-10T19:19:02.387 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo "state": "Replaying" 2023-12-10T19:19:02.387 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep -i Replaying 2023-12-10T19:19:02.388 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "state": "Replaying" 2023-12-10T19:19:02.389 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=started 2023-12-10T19:19:02.389 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test started = started 2023-12-10T19:19:02.389 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:19:02.390 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_49 2023-12-10T19:19:02.390 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:19:02.390 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:02.390 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:02.390 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_49 2023-12-10T19:19:02.390 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = journal ] 2023-12-10T19:19:02.390 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = snapshot ] 2023-12-10T19:19:02.390 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_49 2023-12-10T19:19:02.390 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:19:02.390 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:02.390 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:02.390 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_49 2023-12-10T19:19:02.390 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster2-mirror-image_49.status 2023-12-10T19:19:02.391 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2-mirror-image_49.status 2023-12-10T19:19:02.391 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:19:02.392 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_49.status 2023-12-10T19:19:02.392 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-image_49.status 2023-12-10T19:19:02.392 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-image_49.status 2023-12-10T19:19:02.392 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:19:02.392 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_49.status 2023-12-10T19:19:02.392 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mirror_image_snapshot cluster2 mirror image_49 2023-12-10T19:19:02.392 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:02.392 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:02.392 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_49 2023-12-10T19:19:02.392 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_49 2023-12-10T19:19:02.825 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Snapshot ID: 215 2023-12-10T19:19:02.837 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_49 /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_49.status 2023-12-10T19:19:02.837 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:02.837 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:02.838 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_49 2023-12-10T19:19:02.838 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_49.status 2023-12-10T19:19:02.838 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_49 --format xml 2023-12-10T19:19:02.838 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:19:02.892 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-10T19:19:02.894 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snapshot_id=215 2023-12-10T19:19:02.895 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:19:02.895 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.2 2023-12-10T19:19:03.096 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_49.status 2023-12-10T19:19:03.096 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:19:03.096 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:03.097 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_49 2023-12-10T19:19:03.097 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_49.status 2023-12-10T19:19:03.097 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2023-12-10T19:19:03.097 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:19:03.165 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:19:03.167 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=116 2023-12-10T19:19:03.167 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 215 = 116 2023-12-10T19:19:03.167 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.4 2023-12-10T19:19:03.570 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_49.status 2023-12-10T19:19:03.570 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:19:03.570 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:03.570 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_49 2023-12-10T19:19:03.570 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_49.status 2023-12-10T19:19:03.570 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2023-12-10T19:19:03.570 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:19:03.640 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:19:03.643 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=116 2023-12-10T19:19:03.643 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 215 = 116 2023-12-10T19:19:03.643 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.8 2023-12-10T19:19:04.445 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_49.status 2023-12-10T19:19:04.445 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:19:04.445 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:04.445 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_49 2023-12-10T19:19:04.445 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_49.status 2023-12-10T19:19:04.445 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2023-12-10T19:19:04.445 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:19:04.512 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:19:04.513 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:19:04.514 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:19:04.515 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:19:04.515 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:19:04.515 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:19:04.515 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 215 = 2023-12-10T19:19:04.515 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1.6 2023-12-10T19:19:06.116 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_49.status 2023-12-10T19:19:06.117 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:19:06.117 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:06.117 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_49 2023-12-10T19:19:06.117 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_49.status 2023-12-10T19:19:06.117 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2023-12-10T19:19:06.117 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:19:06.185 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:19:06.187 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=215 2023-12-10T19:19:06.187 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 215 = 215 2023-12-10T19:19:06.188 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:19:06.188 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snap_present cluster1 mirror image_49 snap 2023-12-10T19:19:06.188 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:19:06.188 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:06.188 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_49 2023-12-10T19:19:06.188 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:19:06.188 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:19:06.188 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:19:07.188 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 info mirror/image_49@snap 2023-12-10T19:19:07.249 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_49': 2023-12-10T19:19:07.250 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:19:07.250 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:19:07.250 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 2 2023-12-10T19:19:07.250 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 12bbec4cef00 2023-12-10T19:19:07.250 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.12bbec4cef00 2023-12-10T19:19:07.250 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:19:07.250 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering, non-primary 2023-12-10T19:19:07.250 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:19:07.250 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:19:07.250 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:11:06 2023 2023-12-10T19:19:07.250 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:11:06 2023 2023-12-10T19:19:07.250 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:11:06 2023 2023-12-10T19:19:07.250 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: protected: False 2023-12-10T19:19:07.251 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:19:07.251 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:19:07.251 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: cfaa6e78-26bd-4924-bba0-a389e00cd9fc 2023-12-10T19:19:07.251 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: false 2023-12-10T19:19:07.254 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:19:07.254 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror image_49 snap 2023-12-10T19:19:07.254 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:07.254 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_49 2023-12-10T19:19:07.254 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:19:07.255 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:19:07.255 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_49.export 2023-12-10T19:19:07.255 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_49.export 2023-12-10T19:19:07.255 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_49.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_49.export 2023-12-10T19:19:07.255 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export image_49@snap /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_49.export 2023-12-10T19:19:07.473 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:19:07.483 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export image_49@snap /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_49.export 2023-12-10T19:19:07.875 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:19:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_49.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_49.export 2023-12-10T19:19:07.966 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_49.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_49.export 2023-12-10T19:19:08.007 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:19:08.007 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_50 2023-12-10T19:19:08.007 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_snap cluster2 mirror image_50 snap 2023-12-10T19:19:08.007 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:08.007 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:08.008 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_50 2023-12-10T19:19:08.008 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:19:08.008 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror snap create image_50@snap --debug-rbd=20 --debug-journaler=20 2023-12-10T19:19:08.803 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_started cluster1 mirror image_50 2023-12-10T19:19:08.803 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:19:08.803 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:08.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_50 2023-12-10T19:19:08.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_replay_state cluster1 mirror image_50 started 2023-12-10T19:19:08.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:19:08.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:08.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_50 2023-12-10T19:19:08.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=started 2023-12-10T19:19:08.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:19:08.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:19:09.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_replay_state cluster1 mirror image_50 started 2023-12-10T19:19:09.805 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:19:09.805 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:09.805 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_50 2023-12-10T19:19:09.805 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=started 2023-12-10T19:19:09.805 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_result 2023-12-10T19:19:09.805 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=stopped 2023-12-10T19:19:09.806 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_50 2023-12-10T19:19:09.806 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ + local cluster_instance=cluster1 2023-12-10T19:19:09.806 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:19:09.806 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ greplocal -i state cluster=cluster1 2023-12-10T19:19:09.806 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:19:09.807 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance=cluster1 2023-12-10T19:19:09.807 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loop_instance 2023-12-10T19:19:09.807 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:19:09.807 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 != cluster1 ] 2023-12-10T19:19:09.807 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T19:19:09.808 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_50 2023-12-10T19:19:09.808 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T19:19:09.808 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:19:09.808 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:19:09.809 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T19:19:09.809 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:19:09.809 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:19:09.809 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:19:09.809 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:19:09.809 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:19:09.809 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:19:09.809 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:19:09.809 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:19:09.809 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:19:09.809 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:19:09.809 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-10T19:19:09.809 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:0 2023-12-10T19:19:09.809 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:19:09.809 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:19:09.810 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-10T19:19:09.810 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T19:19:09.810 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:19:09.810 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:19:09.810 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:19:09.810 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:19:09.810 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T19:19:09.810 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:19:09.810 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:19:09.810 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:19:09.810 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:19:09.810 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-10T19:19:09.829 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:19:09.830 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:19:09.830 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-10T19:19:09.830 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_50 2023-12-10T19:19:09.941 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2023-12-10T19:19:09.941 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:19:09.941 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:19:09.943 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_50 2023-12-10T19:19:09.944 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:1 2023-12-10T19:19:09.944 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:19:09.944 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:19:09.944 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-10T19:19:09.944 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:19:09.944 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:19:09.944 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:19:09.944 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:19:09.944 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:19:09.944 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:19:09.944 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:19:09.944 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:19:09.944 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:19:09.944 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:19:09.944 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-10T19:19:09.945 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:1 2023-12-10T19:19:09.945 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:19:09.945 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:19:09.945 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-10T19:19:09.945 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T19:19:09.945 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:19:09.945 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:19:09.945 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:19:09.945 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:19:09.945 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T19:19:09.945 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:19:09.945 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:19:09.945 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T19:19:09.945 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T19:19:09.945 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-10T19:19:09.966 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:19:09.966 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:19:09.967 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-10T19:19:09.967 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_50 2023-12-10T19:19:10.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2023-12-10T19:19:10.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:19:10.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:19:10.078 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_50 2023-12-10T19:19:10.078 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:2 2023-12-10T19:19:10.078 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:19:10.078 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:19:10.078 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-10T19:19:10.078 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:19:10.078 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:19:10.078 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:19:10.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:19:10.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:19:10.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:19:10.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:19:10.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:19:10.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:19:10.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:19:10.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-10T19:19:10.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:2 2023-12-10T19:19:10.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:19:10.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:19:10.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-10T19:19:10.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T19:19:10.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:19:10.080 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:19:10.080 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:19:10.080 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:19:10.080 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T19:19:10.080 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:19:10.080 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:19:10.080 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T19:19:10.080 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T19:19:10.080 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-10T19:19:10.102 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:19:10.103 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:19:10.103 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-10T19:19:10.103 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_50 2023-12-10T19:19:10.216 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2023-12-10T19:19:10.216 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd mirror status : get status for rbd mirror 2023-12-10T19:19:10.216 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:admin_socket: invalid command 2023-12-10T19:19:10.218 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_50 2023-12-10T19:19:10.218 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:3 2023-12-10T19:19:10.218 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:19:10.218 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:19:10.219 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-10T19:19:10.219 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:3 2023-12-10T19:19:10.219 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:19:10.219 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:19:10.219 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:19:10.219 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:19:10.219 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 3 = cluster1:3 ] 2023-12-10T19:19:10.219 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:19:10.219 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:19:10.219 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=3 2023-12-10T19:19:10.219 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:19:10.219 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-10T19:19:10.219 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1:3 2023-12-10T19:19:10.219 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:19:10.220 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:19:10.220 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-10T19:19:10.220 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:3 2023-12-10T19:19:10.220 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=local_cluster 2023-12-10T19:19:10.220 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:19:10.220 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:19:10.220 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:19:10.220 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 3 = cluster1:3 ] 2023-12-10T19:19:10.220 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval local_cluster=cluster1 2023-12-10T19:19:10.220 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local_cluster=cluster1 2023-12-10T19:19:10.220 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=3 2023-12-10T19:19:10.220 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T19:19:10.220 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-10T19:19:10.243 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T19:19:10.243 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T19:19:10.243 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-10T19:19:10.243 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_50 2023-12-10T19:19:10.366 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:19:10.366 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ status_result= "state": "Replaying" 2023-12-10T19:19:10.366 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo "state": "Replaying" 2023-12-10T19:19:10.366 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep -i Replaying 2023-12-10T19:19:10.367 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "state": "Replaying" 2023-12-10T19:19:10.368 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=started 2023-12-10T19:19:10.368 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test started = started 2023-12-10T19:19:10.368 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:19:10.368 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_50 2023-12-10T19:19:10.368 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:19:10.368 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:10.368 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:10.368 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_50 2023-12-10T19:19:10.368 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = journal ] 2023-12-10T19:19:10.368 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ snapshot = snapshot ] 2023-12-10T19:19:10.368 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_50 2023-12-10T19:19:10.368 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_cluster=cluster1 2023-12-10T19:19:10.369 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:10.369 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:10.369 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_50 2023-12-10T19:19:10.369 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster2-mirror-image_50.status 2023-12-10T19:19:10.369 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2-mirror-image_50.status 2023-12-10T19:19:10.369 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:19:10.370 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local status_log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_50.status 2023-12-10T19:19:10.370 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mkfname cluster1-mirror-image_50.status 2023-12-10T19:19:10.370 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1-mirror-image_50.status 2023-12-10T19:19:10.370 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -e s|[/ ]|_|g 2023-12-10T19:19:10.370 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local local_status_log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_50.status 2023-12-10T19:19:10.371 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ mirror_image_snapshot cluster2 mirror image_50 2023-12-10T19:19:10.371 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:10.371 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:10.371 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_50 2023-12-10T19:19:10.371 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_50 2023-12-10T19:19:10.806 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Snapshot ID: 217 2023-12-10T19:19:10.820 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_50 /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_50.status 2023-12-10T19:19:10.821 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:10.821 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:10.821 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_50 2023-12-10T19:19:10.821 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_50.status 2023-12-10T19:19:10.821 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_50 --format xml 2023-12-10T19:19:10.821 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:19:10.875 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-10T19:19:10.878 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snapshot_id=217 2023-12-10T19:19:10.878 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:19:10.878 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.2 2023-12-10T19:19:11.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_50.status 2023-12-10T19:19:11.080 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:19:11.080 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:11.080 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_50 2023-12-10T19:19:11.080 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_50.status 2023-12-10T19:19:11.080 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2023-12-10T19:19:11.080 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:19:11.151 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:19:11.153 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=117 2023-12-10T19:19:11.153 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 217 = 117 2023-12-10T19:19:11.153 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.4 2023-12-10T19:19:11.555 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_50.status 2023-12-10T19:19:11.555 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:19:11.555 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:11.555 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_50 2023-12-10T19:19:11.555 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_50.status 2023-12-10T19:19:11.555 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2023-12-10T19:19:11.555 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:19:11.625 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:19:11.627 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=117 2023-12-10T19:19:11.627 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 217 = 117 2023-12-10T19:19:11.627 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.8 2023-12-10T19:19:12.429 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_50.status 2023-12-10T19:19:12.429 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:19:12.429 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:12.429 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_50 2023-12-10T19:19:12.429 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_50.status 2023-12-10T19:19:12.429 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2023-12-10T19:19:12.429 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:19:12.499 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:19:12.500 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:19:12.503 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:-:1.1: Document is empty 2023-12-10T19:19:12.503 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: 2023-12-10T19:19:12.503 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:^ 2023-12-10T19:19:12.503 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id= 2023-12-10T19:19:12.503 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 217 = 2023-12-10T19:19:12.503 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1.6 2023-12-10T19:19:14.104 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_50.status 2023-12-10T19:19:14.104 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:19:14.104 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:14.104 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_50 2023-12-10T19:19:14.104 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local log=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_50.status 2023-12-10T19:19:14.105 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2023-12-10T19:19:14.105 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-10T19:19:14.166 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-10T19:19:14.169 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local primary_snapshot_id=217 2023-12-10T19:19:14.169 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 217 = 217 2023-12-10T19:19:14.169 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:19:14.169 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_snap_present cluster1 mirror image_50 snap 2023-12-10T19:19:14.169 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:19:14.169 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:14.170 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_50 2023-12-10T19:19:14.170 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:19:14.170 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:19:14.170 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:19:15.170 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 info mirror/image_50@snap 2023-12-10T19:19:15.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_50': 2023-12-10T19:19:15.238 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 128 MiB in 32 objects 2023-12-10T19:19:15.238 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:19:15.238 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 2 2023-12-10T19:19:15.238 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 12bbfd08e488 2023-12-10T19:19:15.238 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.12bbfd08e488 2023-12-10T19:19:15.238 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:19:15.238 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering, non-primary 2023-12-10T19:19:15.238 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:19:15.238 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:19:15.238 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:11:14 2023 2023-12-10T19:19:15.238 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:11:14 2023 2023-12-10T19:19:15.238 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:11:14 2023 2023-12-10T19:19:15.238 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: protected: False 2023-12-10T19:19:15.239 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:19:15.239 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:19:15.239 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: 8d5c864a-1785-4fbb-8bed-7f9006bb0d2b 2023-12-10T19:19:15.239 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: false 2023-12-10T19:19:15.242 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:19:15.242 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ compare_image_snaps mirror image_50 snap 2023-12-10T19:19:15.243 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:15.243 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_50 2023-12-10T19:19:15.243 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap_name=snap 2023-12-10T19:19:15.243 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret=0 2023-12-10T19:19:15.243 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local rmt_export=/tmp/tmp.qNjRyMq75D/cluster2-mirror-image_50.export 2023-12-10T19:19:15.243 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local loc_export=/tmp/tmp.qNjRyMq75D/cluster1-mirror-image_50.export 2023-12-10T19:19:15.243 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_50.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_50.export 2023-12-10T19:19:15.243 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror export image_50@snap /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_50.export 2023-12-10T19:19:15.440 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:19:15.448 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror export image_50@snap /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_50.export 2023-12-10T19:19:15.763 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-10T19:19:15.777 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cmp /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_50.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_50.export 2023-12-10T19:19:15.853 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -f /tmp/tmp.qNjRyMq75D/cluster2-mirror-image_50.export /tmp/tmp.qNjRyMq75D/cluster1-mirror-image_50.export 2023-12-10T19:19:15.886 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:19:15.886 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ testlog TEST: delete many images 2023-12-10T19:19:15.886 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ tee -a /tmp/tmp.qNjRyMq75D/rbd-mirror.test.log 2023-12-10T19:19:15.886 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ date +%F %T 2023-12-10T19:19:15.887 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo 2023-12-10 19:19:15 TEST: delete many images 2023-12-10T19:19:15.887 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:2023-12-10 19:19:15 TEST: delete many images 2023-12-10T19:19:15.887 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 1 50 2023-12-10T19:19:15.888 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_1 2023-12-10T19:19:15.889 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror image_1 snap 2023-12-10T19:19:15.889 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:15.889 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:15.889 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_1 2023-12-10T19:19:15.889 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap 2023-12-10T19:19:15.889 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/image_1@snap 2023-12-10T19:19:16.479 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:19:16.503 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image_retry cluster2 mirror image_1 2023-12-10T19:19:16.503 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:16.503 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:16.503 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_1 2023-12-10T19:19:16.503 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:19:16.504 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_1 2023-12-10T19:19:16.504 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:16.504 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:16.504 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_1 2023-12-10T19:19:16.504 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2023-12-10T19:19:16.619 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2023-12-10T19:19:16.770 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi076.stderr:2023-12-10T19:19:16.779+0000 7f55882ed640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556b57727520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-10T19:19:16.771 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi076.stderr:2023-12-10T19:19:16.779+0000 7f558eafa640 -1 rbd::mirror::ImageReplayer: 0x556b564dd900 [3/8fd23019-e0bb-4243-ad49-4fbc841c8b88] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-10T19:19:17.831 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-10T19:19:17.836 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:19:17.836 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_2 2023-12-10T19:19:17.836 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror image_2 snap 2023-12-10T19:19:17.836 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:17.836 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:17.836 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_2 2023-12-10T19:19:17.836 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap 2023-12-10T19:19:17.836 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/image_2@snap 2023-12-10T19:19:18.499 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:19:18.530 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image_retry cluster2 mirror image_2 2023-12-10T19:19:18.530 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:18.530 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:18.530 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_2 2023-12-10T19:19:18.530 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:19:18.531 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_2 2023-12-10T19:19:18.531 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:18.531 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:18.531 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_2 2023-12-10T19:19:18.531 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2023-12-10T19:19:18.624 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2023-12-10T19:19:18.781 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr:2023-12-10T19:19:18.787+0000 7f2c5add4640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5655570e3380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-10T19:19:18.781 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr:2023-12-10T19:19:18.787+0000 7f2c60de0640 -1 rbd::mirror::ImageReplayer: 0x5655574bca00 [3/13cd2888-c5a1-4f79-8bc5-a118e5cace49] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-10T19:19:19.880 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-10T19:19:19.890 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:19:19.890 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_3 2023-12-10T19:19:19.890 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror image_3 snap 2023-12-10T19:19:19.890 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:19.890 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:19.890 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_3 2023-12-10T19:19:19.890 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap 2023-12-10T19:19:19.890 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/image_3@snap 2023-12-10T19:19:20.492 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:19:20.516 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image_retry cluster2 mirror image_3 2023-12-10T19:19:20.516 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:20.516 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:20.517 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_3 2023-12-10T19:19:20.517 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:19:20.517 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_3 2023-12-10T19:19:20.518 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:20.518 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:20.518 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_3 2023-12-10T19:19:20.518 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2023-12-10T19:19:20.616 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2023-12-10T19:19:20.800 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi076.stderr:2023-12-10T19:19:20.807+0000 7f60851de640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f7b8b20820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-10T19:19:20.800 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi076.stderr:2023-12-10T19:19:20.807+0000 7f608b1ea640 -1 rbd::mirror::ImageReplayer: 0x55f7b7b9c280 [3/3140c852-fad6-4a00-a40c-46869faa3ea1] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-10T19:19:21.816 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-10T19:19:21.826 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:19:21.827 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_4 2023-12-10T19:19:21.827 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror image_4 snap 2023-12-10T19:19:21.827 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:21.827 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:21.827 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_4 2023-12-10T19:19:21.827 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap 2023-12-10T19:19:21.827 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/image_4@snap 2023-12-10T19:19:22.528 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:19:22.545 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image_retry cluster2 mirror image_4 2023-12-10T19:19:22.545 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:22.545 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:22.545 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_4 2023-12-10T19:19:22.545 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:19:22.546 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_4 2023-12-10T19:19:22.546 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:22.546 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:22.546 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_4 2023-12-10T19:19:22.546 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2023-12-10T19:19:22.677 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2023-12-10T19:19:22.848 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr:2023-12-10T19:19:22.855+0000 7f2c5add4640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5655570e3520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-10T19:19:22.848 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr:2023-12-10T19:19:22.855+0000 7f2c60de0640 -1 rbd::mirror::ImageReplayer: 0x5655574bd180 [3/2e46723e-74b1-4e29-b6e7-8617064dd725] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-10T19:19:23.801 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-10T19:19:23.809 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:19:23.809 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_5 2023-12-10T19:19:23.809 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror image_5 snap 2023-12-10T19:19:23.809 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:23.809 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:23.809 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_5 2023-12-10T19:19:23.810 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap 2023-12-10T19:19:23.810 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/image_5@snap 2023-12-10T19:19:24.476 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:19:24.491 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image_retry cluster2 mirror image_5 2023-12-10T19:19:24.492 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:24.492 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:24.492 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_5 2023-12-10T19:19:24.492 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:19:24.492 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_5 2023-12-10T19:19:24.492 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:24.492 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:24.493 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_5 2023-12-10T19:19:24.493 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2023-12-10T19:19:24.583 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2023-12-10T19:19:24.909 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi076.stderr:2023-12-10T19:19:24.915+0000 7f7f8e8d6640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562af608ad00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-10T19:19:24.909 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi076.stderr:2023-12-10T19:19:24.915+0000 7f7f948e2640 -1 rbd::mirror::ImageReplayer: 0x562af5484280 [3/c911f848-6f73-4853-a1d8-243e89af66d6] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-10T19:19:25.828 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-10T19:19:25.832 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:19:25.832 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_6 2023-12-10T19:19:25.832 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror image_6 snap 2023-12-10T19:19:25.832 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:25.832 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:25.832 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_6 2023-12-10T19:19:25.833 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap 2023-12-10T19:19:25.833 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/image_6@snap 2023-12-10T19:19:26.592 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:19:26.620 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image_retry cluster2 mirror image_6 2023-12-10T19:19:26.620 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:26.620 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:26.620 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_6 2023-12-10T19:19:26.620 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:19:26.621 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_6 2023-12-10T19:19:26.621 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:26.621 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:26.621 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_6 2023-12-10T19:19:26.621 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2023-12-10T19:19:26.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2023-12-10T19:19:27.125 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi076.stderr:2023-12-10T19:19:27.130+0000 7f55882ed640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556b577f9a00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-10T19:19:27.125 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi076.stderr:2023-12-10T19:19:27.130+0000 7f558eafa640 -1 rbd::mirror::ImageReplayer: 0x556b564dd400 [3/c1f7c734-b9ec-4996-95de-10380a5d720f] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-10T19:19:27.929 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-10T19:19:27.935 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:19:27.935 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_7 2023-12-10T19:19:27.935 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror image_7 snap 2023-12-10T19:19:27.935 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:27.935 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:27.935 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_7 2023-12-10T19:19:27.935 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap 2023-12-10T19:19:27.935 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/image_7@snap 2023-12-10T19:19:28.577 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:19:28.590 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image_retry cluster2 mirror image_7 2023-12-10T19:19:28.590 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:28.590 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:28.591 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_7 2023-12-10T19:19:28.591 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:19:28.591 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_7 2023-12-10T19:19:28.591 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:28.592 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:28.592 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_7 2023-12-10T19:19:28.592 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2023-12-10T19:19:28.716 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2023-12-10T19:19:28.956 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi076.stderr:2023-12-10T19:19:28.962+0000 7f7f8e0d5640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562af608a4e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-10T19:19:28.957 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi076.stderr:2023-12-10T19:19:28.962+0000 7f7f948e2640 -1 rbd::mirror::ImageReplayer: 0x562af5e16a00 [3/f8539d7e-11cd-4f60-9d51-e1da51589906] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-10T19:19:29.771 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-10T19:19:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:19:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_8 2023-12-10T19:19:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror image_8 snap 2023-12-10T19:19:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_8 2023-12-10T19:19:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap 2023-12-10T19:19:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/image_8@snap 2023-12-10T19:19:30.594 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:19:30.607 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image_retry cluster2 mirror image_8 2023-12-10T19:19:30.607 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:30.607 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:30.607 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_8 2023-12-10T19:19:30.607 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:19:30.608 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_8 2023-12-10T19:19:30.608 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:30.608 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:30.608 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_8 2023-12-10T19:19:30.608 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2023-12-10T19:19:30.733 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2023-12-10T19:19:30.958 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi076.stderr:2023-12-10T19:19:30.966+0000 7f5588aee640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556b57727860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-10T19:19:30.959 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi076.stderr:2023-12-10T19:19:30.966+0000 7f558eafa640 -1 rbd::mirror::ImageReplayer: 0x556b57534c80 [3/f24c77de-98cd-45a4-9355-6e2e919e8517] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-10T19:19:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-10T19:19:31.962 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:19:31.962 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_9 2023-12-10T19:19:31.962 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror image_9 snap 2023-12-10T19:19:31.962 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:31.962 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:31.962 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_9 2023-12-10T19:19:31.962 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap 2023-12-10T19:19:31.962 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/image_9@snap 2023-12-10T19:19:32.609 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:19:32.653 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image_retry cluster2 mirror image_9 2023-12-10T19:19:32.653 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:32.653 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:32.653 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_9 2023-12-10T19:19:32.653 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:19:32.653 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_9 2023-12-10T19:19:32.653 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:32.654 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:32.654 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_9 2023-12-10T19:19:32.654 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2023-12-10T19:19:32.724 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2023-12-10T19:19:32.929 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi076.stderr:2023-12-10T19:19:32.934+0000 7f7f8e0d5640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562af608a680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-10T19:19:32.929 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi076.stderr:2023-12-10T19:19:32.934+0000 7f7f948e2640 -1 rbd::mirror::ImageReplayer: 0x562af5484500 [3/ab1975bd-7976-4b27-85c2-606054317fe6] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-10T19:19:33.891 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-10T19:19:33.897 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:19:33.897 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_10 2023-12-10T19:19:33.897 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror image_10 snap 2023-12-10T19:19:33.898 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:33.898 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:33.898 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_10 2023-12-10T19:19:33.898 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap 2023-12-10T19:19:33.898 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/image_10@snap 2023-12-10T19:19:34.510 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:19:34.532 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image_retry cluster2 mirror image_10 2023-12-10T19:19:34.533 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:34.533 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:34.533 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_10 2023-12-10T19:19:34.533 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:19:34.534 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_10 2023-12-10T19:19:34.534 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:34.534 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:34.534 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_10 2023-12-10T19:19:34.534 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2023-12-10T19:19:34.645 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2023-12-10T19:19:34.832 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi076.stderr:2023-12-10T19:19:34.838+0000 7f7f8e8d6640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562af65681a0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-10T19:19:34.832 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi076.stderr:2023-12-10T19:19:34.838+0000 7f7f950e3640 -1 rbd::mirror::ImageReplayer: 0x562af5484f00 [3/5e79b28d-acd2-4fbc-9a84-6c7b4c7dc476] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-10T19:19:36.031 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-10T19:19:36.035 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:19:36.035 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_11 2023-12-10T19:19:36.035 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror image_11 snap 2023-12-10T19:19:36.035 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:36.035 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:36.035 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_11 2023-12-10T19:19:36.035 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap 2023-12-10T19:19:36.035 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/image_11@snap 2023-12-10T19:19:36.642 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:19:36.659 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image_retry cluster2 mirror image_11 2023-12-10T19:19:36.659 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:36.659 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_11 2023-12-10T19:19:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:19:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_11 2023-12-10T19:19:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_11 2023-12-10T19:19:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_11 2023-12-10T19:19:36.772 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_11 2023-12-10T19:19:37.052 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi076.stderr:2023-12-10T19:19:37.058+0000 7f55882ed640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556b57727a00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-10T19:19:37.052 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi076.stderr:2023-12-10T19:19:37.058+0000 7f558f2fb640 -1 rbd::mirror::ImageReplayer: 0x556b564ddb80 [3/d372b711-4545-4edb-a533-b051b75e68e3] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-10T19:19:37.935 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-10T19:19:37.940 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:19:37.940 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_12 2023-12-10T19:19:37.940 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror image_12 snap 2023-12-10T19:19:37.940 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:37.940 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:37.940 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_12 2023-12-10T19:19:37.940 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap 2023-12-10T19:19:37.940 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/image_12@snap 2023-12-10T19:19:38.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:19:38.691 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image_retry cluster2 mirror image_12 2023-12-10T19:19:38.691 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:38.692 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:38.692 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_12 2023-12-10T19:19:38.692 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:19:38.692 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_12 2023-12-10T19:19:38.692 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:38.692 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:38.692 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_12 2023-12-10T19:19:38.692 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_12 2023-12-10T19:19:38.779 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_12 2023-12-10T19:19:38.956 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi076.stderr:2023-12-10T19:19:38.962+0000 7f60849dd640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f7b8b209c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-10T19:19:38.957 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi076.stderr:2023-12-10T19:19:38.962+0000 7f608b9eb640 -1 rbd::mirror::ImageReplayer: 0x55f7b7b9cc80 [3/f2b859e8-6dbd-4781-91cf-11df1eb8a7a0] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-10T19:19:39.843 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-10T19:19:39.847 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:19:39.848 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_13 2023-12-10T19:19:39.848 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror image_13 snap 2023-12-10T19:19:39.848 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:39.848 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:39.848 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_13 2023-12-10T19:19:39.848 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap 2023-12-10T19:19:39.848 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/image_13@snap 2023-12-10T19:19:40.688 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:19:40.712 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image_retry cluster2 mirror image_13 2023-12-10T19:19:40.712 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:40.712 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:40.712 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_13 2023-12-10T19:19:40.712 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:19:40.718 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_13 2023-12-10T19:19:40.718 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:40.718 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:40.718 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_13 2023-12-10T19:19:40.718 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_13 2023-12-10T19:19:40.835 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_13 2023-12-10T19:19:40.911 INFO:tasks.ceph.cluster1.osd.1.smithi033.stderr:2023-12-10T19:19:40.903+0000 7f06a5518640 -1 reset not still connected to 0x563fc2f6dd40 2023-12-10T19:19:40.911 INFO:tasks.ceph.cluster1.osd.1.smithi033.stderr:2023-12-10T19:19:40.907+0000 7f06a5518640 -1 reset not still connected to 0x563fc4818c30 2023-12-10T19:19:41.063 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi076.stderr:2023-12-10T19:19:41.070+0000 7f60849dd640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f7b8ae91e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-10T19:19:41.063 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi076.stderr:2023-12-10T19:19:41.070+0000 7f608b9eb640 -1 rbd::mirror::ImageReplayer: 0x55f7b7b9c780 [3/9cd9cf4d-71e8-4901-8340-6c85f5a313e8] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-10T19:19:42.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-10T19:19:42.027 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:19:42.028 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_14 2023-12-10T19:19:42.028 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror image_14 snap 2023-12-10T19:19:42.028 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:42.028 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:42.028 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_14 2023-12-10T19:19:42.028 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap 2023-12-10T19:19:42.028 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/image_14@snap 2023-12-10T19:19:42.701 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:19:42.713 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image_retry cluster2 mirror image_14 2023-12-10T19:19:42.713 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:42.713 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:42.713 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_14 2023-12-10T19:19:42.713 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:19:42.714 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_14 2023-12-10T19:19:42.714 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:42.714 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:42.714 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_14 2023-12-10T19:19:42.714 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_14 2023-12-10T19:19:42.822 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_14 2023-12-10T19:19:42.980 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr:2023-12-10T19:19:42.986+0000 7f2c5add4640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5655570e36c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-10T19:19:42.980 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr:2023-12-10T19:19:42.986+0000 7f2c60de0640 -1 rbd::mirror::ImageReplayer: 0x5655574bcf00 [3/1d54e641-de62-4511-9abb-01dd3218783c] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-10T19:19:43.968 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-10T19:19:43.972 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:19:43.972 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_15 2023-12-10T19:19:43.972 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror image_15 snap 2023-12-10T19:19:43.973 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:43.973 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:43.973 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_15 2023-12-10T19:19:43.973 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap 2023-12-10T19:19:43.973 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/image_15@snap 2023-12-10T19:19:44.512 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:19:44.529 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image_retry cluster2 mirror image_15 2023-12-10T19:19:44.530 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:44.530 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:44.530 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_15 2023-12-10T19:19:44.530 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:19:44.530 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_15 2023-12-10T19:19:44.530 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:44.531 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:44.531 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_15 2023-12-10T19:19:44.531 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_15 2023-12-10T19:19:44.594 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_15 2023-12-10T19:19:45.061 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi076.stderr:2023-12-10T19:19:45.066+0000 7f60849dd640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f7b8b20b60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-10T19:19:45.076 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi076.stderr:2023-12-10T19:19:45.082+0000 7f60849dd640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f7b8ae9380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-10T19:19:45.076 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi076.stderr:2023-12-10T19:19:45.082+0000 7f608b9eb640 -1 rbd::mirror::ImageReplayer: 0x55f7b7b9ca00 [3/e5868311-cdb8-441b-a90b-45641cca9efd] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-10T19:19:46.102 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-10T19:19:46.108 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:19:46.108 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_16 2023-12-10T19:19:46.108 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror image_16 snap 2023-12-10T19:19:46.108 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:46.108 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:46.108 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_16 2023-12-10T19:19:46.108 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap 2023-12-10T19:19:46.108 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/image_16@snap 2023-12-10T19:19:46.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:19:46.739 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image_retry cluster2 mirror image_16 2023-12-10T19:19:46.739 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:46.739 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:46.739 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_16 2023-12-10T19:19:46.739 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:19:46.740 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_16 2023-12-10T19:19:46.740 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:46.740 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:46.740 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_16 2023-12-10T19:19:46.740 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_16 2023-12-10T19:19:46.834 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_16 2023-12-10T19:19:47.118 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi076.stderr:2023-12-10T19:19:47.121+0000 7f60849dd640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f7b8b20d00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-10T19:19:47.118 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi076.stderr:2023-12-10T19:19:47.121+0000 7f608b1ea640 -1 rbd::mirror::ImageReplayer: 0x55f7b7b9d900 [3/66ef9828-72b1-4f4f-b6ef-90d776f972ce] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-10T19:19:48.009 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-10T19:19:48.021 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:19:48.021 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_17 2023-12-10T19:19:48.021 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror image_17 snap 2023-12-10T19:19:48.021 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:48.021 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:48.021 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_17 2023-12-10T19:19:48.021 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap 2023-12-10T19:19:48.021 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/image_17@snap 2023-12-10T19:19:48.813 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:19:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image_retry cluster2 mirror image_17 2023-12-10T19:19:48.829 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:48.829 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:48.829 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_17 2023-12-10T19:19:48.829 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:19:48.830 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_17 2023-12-10T19:19:48.830 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:48.830 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:48.830 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_17 2023-12-10T19:19:48.830 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_17 2023-12-10T19:19:48.910 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_17 2023-12-10T19:19:49.074 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi076.stderr:2023-12-10T19:19:49.081+0000 7f5588aee640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556b577f9ba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-10T19:19:49.075 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi076.stderr:2023-12-10T19:19:49.081+0000 7f558f2fb640 -1 rbd::mirror::ImageReplayer: 0x556b57534500 [3/a44b2376-7333-4134-9c68-c71acef2b46e] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-10T19:19:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-10T19:19:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:19:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_18 2023-12-10T19:19:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror image_18 snap 2023-12-10T19:19:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:49.882 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:49.882 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_18 2023-12-10T19:19:49.882 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap 2023-12-10T19:19:49.882 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/image_18@snap 2023-12-10T19:19:50.770 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:19:50.784 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image_retry cluster2 mirror image_18 2023-12-10T19:19:50.784 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:50.784 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:50.784 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_18 2023-12-10T19:19:50.784 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:19:50.785 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_18 2023-12-10T19:19:50.786 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:50.786 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:50.786 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_18 2023-12-10T19:19:50.786 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_18 2023-12-10T19:19:50.888 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_18 2023-12-10T19:19:51.152 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr:2023-12-10T19:19:51.157+0000 7f2c5add4640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x565557127040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-10T19:19:51.153 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr:2023-12-10T19:19:51.157+0000 7f2c615e1640 -1 rbd::mirror::ImageReplayer: 0x5655574bc500 [3/013b5e35-24cb-486f-93bf-0f941ebb1c1d] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-10T19:19:51.843 INFO:tasks.ceph.cluster2.osd.2.smithi076.stderr:2023-12-10T19:19:51.845+0000 7f7f6add3640 -1 reset not still connected to 0x56430d0a3ba0 2023-12-10T19:19:51.845 INFO:tasks.ceph.cluster2.osd.2.smithi076.stderr:2023-12-10T19:19:51.849+0000 7f7f6add3640 -1 reset not still connected to 0x564318e95a00 2023-12-10T19:19:51.845 INFO:tasks.ceph.cluster2.osd.2.smithi076.stderr:2023-12-10T19:19:51.849+0000 7f7f6add3640 -1 reset not still connected to 0x56432fbfd930 2023-12-10T19:19:52.050 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-10T19:19:52.055 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:19:52.055 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_19 2023-12-10T19:19:52.056 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror image_19 snap 2023-12-10T19:19:52.056 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:52.056 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:52.056 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_19 2023-12-10T19:19:52.056 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap 2023-12-10T19:19:52.056 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/image_19@snap 2023-12-10T19:19:52.847 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:19:52.869 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image_retry cluster2 mirror image_19 2023-12-10T19:19:52.869 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:52.869 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:52.869 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_19 2023-12-10T19:19:52.869 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:19:52.883 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_19 2023-12-10T19:19:52.883 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:52.883 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:52.883 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_19 2023-12-10T19:19:52.883 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_19 2023-12-10T19:19:52.976 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_19 2023-12-10T19:19:53.124 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi076.stderr:2023-12-10T19:19:53.129+0000 7f60851de640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f7b8ae9520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-10T19:19:53.124 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi076.stderr:2023-12-10T19:19:53.129+0000 7f608b9eb640 -1 rbd::mirror::ImageReplayer: 0x55f7b7b9c500 [3/9885ccb8-b168-4992-a2b7-1fe906e83484] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-10T19:19:54.232 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-10T19:19:54.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:19:54.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_20 2023-12-10T19:19:54.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror image_20 snap 2023-12-10T19:19:54.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:54.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:54.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_20 2023-12-10T19:19:54.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap 2023-12-10T19:19:54.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/image_20@snap 2023-12-10T19:19:54.522 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:19:54.531 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image_retry cluster2 mirror image_20 2023-12-10T19:19:54.531 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:54.531 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:54.531 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_20 2023-12-10T19:19:54.531 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:19:54.531 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_20 2023-12-10T19:19:54.532 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:54.532 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:54.532 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_20 2023-12-10T19:19:54.532 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_20 2023-12-10T19:19:54.598 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_20 2023-12-10T19:19:54.782 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi076.stderr:2023-12-10T19:19:54.785+0000 7f60849dd640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f7b8b20ea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-10T19:19:54.782 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi076.stderr:2023-12-10T19:19:54.785+0000 7f608b1ea640 -1 rbd::mirror::ImageReplayer: 0x55f7b7b9db80 [3/ad3d186e-bfbd-40cc-b37d-73be0b0f0a7d] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-10T19:19:55.048 INFO:tasks.ceph.cluster1.osd.2.smithi033.stderr:2023-12-10T19:19:55.043+0000 7fef8ab8e640 -1 reset not still connected to 0x55c5a1a855f0 2023-12-10T19:19:55.050 INFO:tasks.ceph.cluster1.osd.2.smithi033.stderr:2023-12-10T19:19:55.047+0000 7fef8ab8e640 -1 reset not still connected to 0x55c5a1a85790 2023-12-10T19:19:55.050 INFO:tasks.ceph.cluster1.osd.2.smithi033.stderr:2023-12-10T19:19:55.047+0000 7fef8ab8e640 -1 reset not still connected to 0x55c5a1a85a00 2023-12-10T19:19:55.050 INFO:tasks.ceph.cluster1.osd.2.smithi033.stderr:2023-12-10T19:19:55.047+0000 7fef8ab8e640 -1 reset not still connected to 0x55c5a20ee0d0 2023-12-10T19:19:55.050 INFO:tasks.ceph.cluster1.osd.2.smithi033.stderr:2023-12-10T19:19:55.047+0000 7fef8ab8e640 -1 reset not still connected to 0x55c5a20ef380 2023-12-10T19:19:55.050 INFO:tasks.ceph.cluster1.osd.2.smithi033.stderr:2023-12-10T19:19:55.047+0000 7fef8ab8e640 -1 reset not still connected to 0x55c5a385b5f0 2023-12-10T19:19:55.050 INFO:tasks.ceph.cluster1.osd.2.smithi033.stderr:2023-12-10T19:19:55.047+0000 7fef8ab8e640 -1 reset not still connected to 0x55c5a4438270 2023-12-10T19:19:55.050 INFO:tasks.ceph.cluster1.osd.2.smithi033.stderr:2023-12-10T19:19:55.047+0000 7fef8ab8e640 -1 reset not still connected to 0x55c5a4ce80d0 2023-12-10T19:19:55.121 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi076.stderr:2023-12-10T19:19:55.125+0000 7f60851de640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f7b8b21380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-10T19:19:55.121 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi076.stderr:2023-12-10T19:19:55.125+0000 7f608b9eb640 -1 rbd::mirror::ImageReplayer: 0x55f7b7b9db80 [3/ad3d186e-bfbd-40cc-b37d-73be0b0f0a7d] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-10T19:19:55.835 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-10T19:19:55.838 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:19:55.839 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_21 2023-12-10T19:19:55.839 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror image_21 snap 2023-12-10T19:19:55.839 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:55.839 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:55.839 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_21 2023-12-10T19:19:55.839 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap 2023-12-10T19:19:55.839 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/image_21@snap 2023-12-10T19:19:56.826 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:19:56.843 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image_retry cluster2 mirror image_21 2023-12-10T19:19:56.843 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:56.843 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:56.843 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_21 2023-12-10T19:19:56.843 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:19:56.844 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_21 2023-12-10T19:19:56.844 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:56.844 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:56.844 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_21 2023-12-10T19:19:56.844 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_21 2023-12-10T19:19:56.966 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_21 2023-12-10T19:19:57.247 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr:2023-12-10T19:19:57.253+0000 7f2c5a5d3640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5655571271e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-10T19:19:57.247 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr:2023-12-10T19:19:57.253+0000 7f2c615e1640 -1 rbd::mirror::ImageReplayer: 0x565556ff0280 [3/341379c3-72c1-42e0-8bda-bc8752bf5c19] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-10T19:19:58.211 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-10T19:19:58.215 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:19:58.215 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_22 2023-12-10T19:19:58.215 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror image_22 snap 2023-12-10T19:19:58.215 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:58.215 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:58.215 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_22 2023-12-10T19:19:58.215 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap 2023-12-10T19:19:58.215 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/image_22@snap 2023-12-10T19:19:58.835 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:19:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image_retry cluster2 mirror image_22 2023-12-10T19:19:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_22 2023-12-10T19:19:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:19:58.848 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_22 2023-12-10T19:19:58.848 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:58.848 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:58.848 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_22 2023-12-10T19:19:58.848 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_22 2023-12-10T19:19:58.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_22 2023-12-10T19:19:59.159 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi076.stderr:2023-12-10T19:19:59.165+0000 7f7f8e8d6640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562af5b0cb60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-10T19:19:59.159 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi076.stderr:2023-12-10T19:19:59.165+0000 7f7f948e2640 -1 rbd::mirror::ImageReplayer: 0x562af5485180 [3/5f3f027e-9f9a-463e-be32-1e7c54f42e4e] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-10T19:19:59.160 INFO:tasks.ceph.cluster1.osd.2.smithi033.stderr:2023-12-10T19:19:59.159+0000 7fef8ab8e640 -1 reset not still connected to 0x55c59dfb1860 2023-12-10T19:19:59.160 INFO:tasks.ceph.cluster1.osd.2.smithi033.stderr:2023-12-10T19:19:59.159+0000 7fef8ab8e640 -1 reset not still connected to 0x55c59efc44e0 2023-12-10T19:19:59.160 INFO:tasks.ceph.cluster1.osd.2.smithi033.stderr:2023-12-10T19:19:59.159+0000 7fef8ab8e640 -1 reset not still connected to 0x55c5a11d44e0 2023-12-10T19:19:59.160 INFO:tasks.ceph.cluster1.osd.2.smithi033.stderr:2023-12-10T19:19:59.159+0000 7fef8ab8e640 -1 reset not still connected to 0x55c5a4438680 2023-12-10T19:19:59.161 INFO:tasks.ceph.cluster1.osd.2.smithi033.stderr:2023-12-10T19:19:59.159+0000 7fef8ab8e640 -1 reset not still connected to 0x55c5a4b12340 2023-12-10T19:19:59.828 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-10T19:19:59.840 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:19:59.840 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_23 2023-12-10T19:19:59.840 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror image_23 snap 2023-12-10T19:19:59.840 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:19:59.841 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:19:59.841 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_23 2023-12-10T19:19:59.841 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap 2023-12-10T19:19:59.841 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/image_23@snap 2023-12-10T19:20:00.875 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:20:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image_retry cluster2 mirror image_23 2023-12-10T19:20:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_23 2023-12-10T19:20:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:20:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_23 2023-12-10T19:20:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_23 2023-12-10T19:20:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_23 2023-12-10T19:20:01.170 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_23 2023-12-10T19:20:01.301 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi076.stderr:2023-12-10T19:20:01.305+0000 7f60851de640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f7b8ae96c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-10T19:20:01.301 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi076.stderr:2023-12-10T19:20:01.305+0000 7f608b9eb640 -1 rbd::mirror::ImageReplayer: 0x55f7b8552500 [3/b1ff34f6-63f9-49d9-9315-4eabc97c393e] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-10T19:20:02.133 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-10T19:20:02.143 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:20:02.143 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_24 2023-12-10T19:20:02.143 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror image_24 snap 2023-12-10T19:20:02.143 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:02.143 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:02.143 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_24 2023-12-10T19:20:02.143 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap 2023-12-10T19:20:02.143 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/image_24@snap 2023-12-10T19:20:02.910 INFO:tasks.ceph.cluster1.osd.0.smithi033.stderr:2023-12-10T19:20:02.911+0000 7f836f941640 -1 reset not still connected to 0x558da14a3ba0 2023-12-10T19:20:02.911 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:20:02.949 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image_retry cluster2 mirror image_24 2023-12-10T19:20:02.949 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:02.949 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:02.949 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_24 2023-12-10T19:20:02.949 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:20:02.950 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_24 2023-12-10T19:20:02.950 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:02.950 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:02.950 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_24 2023-12-10T19:20:02.950 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_24 2023-12-10T19:20:03.067 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_24 2023-12-10T19:20:03.218 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr:2023-12-10T19:20:03.221+0000 7f2c5add4640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5655570e3380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-10T19:20:03.218 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr:2023-12-10T19:20:03.221+0000 7f2c60de0640 -1 rbd::mirror::ImageReplayer: 0x565556ff0000 [3/2920a7c1-c4ea-4349-87ad-88428404bcc1] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-10T19:20:04.268 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-10T19:20:04.273 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:20:04.273 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_25 2023-12-10T19:20:04.273 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror image_25 snap 2023-12-10T19:20:04.273 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:04.273 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:04.273 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_25 2023-12-10T19:20:04.273 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap 2023-12-10T19:20:04.273 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/image_25@snap 2023-12-10T19:20:05.522 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:20:05.540 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image_retry cluster2 mirror image_25 2023-12-10T19:20:05.540 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_25 2023-12-10T19:20:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:20:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_25 2023-12-10T19:20:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_25 2023-12-10T19:20:05.542 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_25 2023-12-10T19:20:05.613 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_25 2023-12-10T19:20:05.823 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr:2023-12-10T19:20:05.825+0000 7f2c5a5d3640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5655570d8d00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-10T19:20:05.823 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr:2023-12-10T19:20:05.825+0000 7f2c615e1640 -1 rbd::mirror::ImageReplayer: 0x565556ff0780 [3/cd328d00-062e-4168-a2e8-c67acd6bae34] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-10T19:20:06.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-10T19:20:06.817 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:20:06.818 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_26 2023-12-10T19:20:06.818 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror image_26 snap 2023-12-10T19:20:06.818 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:06.818 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:06.818 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_26 2023-12-10T19:20:06.819 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap 2023-12-10T19:20:06.819 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/image_26@snap 2023-12-10T19:20:07.543 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:20:07.556 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image_retry cluster2 mirror image_26 2023-12-10T19:20:07.556 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:07.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:07.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_26 2023-12-10T19:20:07.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:20:07.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_26 2023-12-10T19:20:07.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:07.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:07.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_26 2023-12-10T19:20:07.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_26 2023-12-10T19:20:07.652 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_26 2023-12-10T19:20:07.845 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi076.stderr:2023-12-10T19:20:07.848+0000 7f55882ed640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556b576fd6c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-10T19:20:07.846 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi076.stderr:2023-12-10T19:20:07.848+0000 7f558eafa640 -1 rbd::mirror::ImageReplayer: 0x556b564dca00 [3/b6b891c6-1bf3-4053-93b5-065fbe9ec26b] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-10T19:20:08.819 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-10T19:20:08.824 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:20:08.824 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_27 2023-12-10T19:20:08.824 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror image_27 snap 2023-12-10T19:20:08.824 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:08.824 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:08.824 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_27 2023-12-10T19:20:08.824 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap 2023-12-10T19:20:08.824 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/image_27@snap 2023-12-10T19:20:09.505 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:20:09.518 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image_retry cluster2 mirror image_27 2023-12-10T19:20:09.519 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:09.519 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:09.519 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_27 2023-12-10T19:20:09.519 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:20:09.519 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_27 2023-12-10T19:20:09.519 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:09.520 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:09.520 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_27 2023-12-10T19:20:09.520 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_27 2023-12-10T19:20:09.646 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_27 2023-12-10T19:20:09.944 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi076.stderr:2023-12-10T19:20:09.948+0000 7f5588aee640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556b577f9d40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-10T19:20:09.944 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi076.stderr:2023-12-10T19:20:09.948+0000 7f558f2fb640 -1 rbd::mirror::ImageReplayer: 0x556b564dd680 [3/ef2fa6c2-fc2c-4d0b-9dde-4392f17b097a] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-10T19:20:10.842 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-10T19:20:10.847 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:20:10.847 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_28 2023-12-10T19:20:10.847 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror image_28 snap 2023-12-10T19:20:10.847 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:10.847 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:10.847 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_28 2023-12-10T19:20:10.848 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap 2023-12-10T19:20:10.848 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/image_28@snap 2023-12-10T19:20:11.568 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:20:11.584 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image_retry cluster2 mirror image_28 2023-12-10T19:20:11.584 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:11.584 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:11.584 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_28 2023-12-10T19:20:11.584 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:20:11.584 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_28 2023-12-10T19:20:11.585 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:11.585 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:11.585 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_28 2023-12-10T19:20:11.585 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_28 2023-12-10T19:20:11.699 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_28 2023-12-10T19:20:11.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi076.stderr:2023-12-10T19:20:11.956+0000 7f55882ed640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556b57503040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-10T19:20:11.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi076.stderr:2023-12-10T19:20:11.956+0000 7f558f2fb640 -1 rbd::mirror::ImageReplayer: 0x556b564dc780 [3/8f8a9063-1808-4763-bb98-81cd23d63bba] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-10T19:20:12.962 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-10T19:20:12.967 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:20:12.967 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_29 2023-12-10T19:20:12.968 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror image_29 snap 2023-12-10T19:20:12.968 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:12.968 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:12.968 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_29 2023-12-10T19:20:12.968 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap 2023-12-10T19:20:12.968 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/image_29@snap 2023-12-10T19:20:13.592 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:20:13.603 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image_retry cluster2 mirror image_29 2023-12-10T19:20:13.603 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:13.603 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:13.603 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_29 2023-12-10T19:20:13.604 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:20:13.604 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_29 2023-12-10T19:20:13.604 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:13.604 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:13.604 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_29 2023-12-10T19:20:13.604 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_29 2023-12-10T19:20:13.724 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_29 2023-12-10T19:20:13.990 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi076.stderr:2023-12-10T19:20:13.992+0000 7f7f8e0d5640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562af5b0c9c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-10T19:20:13.990 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi076.stderr:2023-12-10T19:20:13.992+0000 7f7f948e2640 -1 rbd::mirror::ImageReplayer: 0x562af5e16c80 [3/dc1ce313-d80e-4cd6-a4dd-74802559481e] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-10T19:20:14.780 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-10T19:20:14.791 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:20:14.791 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_30 2023-12-10T19:20:14.791 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror image_30 snap 2023-12-10T19:20:14.791 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:14.792 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:14.792 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_30 2023-12-10T19:20:14.792 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap 2023-12-10T19:20:14.792 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/image_30@snap 2023-12-10T19:20:15.610 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:20:15.622 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image_retry cluster2 mirror image_30 2023-12-10T19:20:15.622 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:15.622 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:15.622 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_30 2023-12-10T19:20:15.622 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:20:15.623 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_30 2023-12-10T19:20:15.623 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:15.623 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:15.623 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_30 2023-12-10T19:20:15.623 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_30 2023-12-10T19:20:15.743 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_30 2023-12-10T19:20:15.975 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi076.stderr:2023-12-10T19:20:15.980+0000 7f60849dd640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f7b8ae9860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-10T19:20:15.975 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi076.stderr:2023-12-10T19:20:15.980+0000 7f608b9eb640 -1 rbd::mirror::ImageReplayer: 0x55f7b8552280 [3/4d4eb965-6ed2-4ccb-a76d-8e89ea006260] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-10T19:20:16.907 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-10T19:20:16.915 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:20:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_31 2023-12-10T19:20:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror image_31 snap 2023-12-10T19:20:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:16.918 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:16.918 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_31 2023-12-10T19:20:16.918 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap 2023-12-10T19:20:16.918 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/image_31@snap 2023-12-10T19:20:17.635 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:20:17.659 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image_retry cluster2 mirror image_31 2023-12-10T19:20:17.659 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:17.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:17.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_31 2023-12-10T19:20:17.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:20:17.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_31 2023-12-10T19:20:17.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:17.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:17.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_31 2023-12-10T19:20:17.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_31 2023-12-10T19:20:17.768 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_31 2023-12-10T19:20:17.958 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi076.stderr:2023-12-10T19:20:17.960+0000 7f7f8e0d5640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562af5a3bd40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-10T19:20:17.959 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi076.stderr:2023-12-10T19:20:17.960+0000 7f7f950e3640 -1 rbd::mirror::ImageReplayer: 0x562af5484780 [3/899d2f20-51d9-4c49-ba24-b07293c27322] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-10T19:20:19.020 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-10T19:20:19.024 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:20:19.024 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_32 2023-12-10T19:20:19.024 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror image_32 snap 2023-12-10T19:20:19.024 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:19.024 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:19.024 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_32 2023-12-10T19:20:19.024 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap 2023-12-10T19:20:19.024 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/image_32@snap 2023-12-10T19:20:19.547 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:20:19.558 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image_retry cluster2 mirror image_32 2023-12-10T19:20:19.558 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:19.558 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:19.558 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_32 2023-12-10T19:20:19.558 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:20:19.564 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_32 2023-12-10T19:20:19.564 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:19.564 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:19.564 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_32 2023-12-10T19:20:19.564 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_32 2023-12-10T19:20:19.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_32 2023-12-10T19:20:19.890 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi076.stderr:2023-12-10T19:20:19.892+0000 7f7f8e8d6640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562af5b0cd00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-10T19:20:19.891 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi076.stderr:2023-12-10T19:20:19.892+0000 7f7f948e2640 -1 rbd::mirror::ImageReplayer: 0x562af5484a00 [3/f5919ca9-8590-41d3-8074-153a93205199] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-10T19:20:19.913 INFO:tasks.ceph.cluster1.osd.0.smithi033.stderr:2023-12-10T19:20:19.914+0000 7f836f941640 -1 reset not still connected to 0x558da14a3ba0 2023-12-10T19:20:19.913 INFO:tasks.ceph.cluster1.osd.0.smithi033.stderr:2023-12-10T19:20:19.914+0000 7f836f941640 -1 reset not still connected to 0x558da168dee0 2023-12-10T19:20:21.029 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-10T19:20:21.032 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:20:21.032 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_33 2023-12-10T19:20:21.032 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror image_33 snap 2023-12-10T19:20:21.032 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:21.032 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:21.033 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_33 2023-12-10T19:20:21.033 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap 2023-12-10T19:20:21.033 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/image_33@snap 2023-12-10T19:20:21.665 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:20:21.679 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image_retry cluster2 mirror image_33 2023-12-10T19:20:21.679 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:21.680 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:21.680 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_33 2023-12-10T19:20:21.680 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:20:21.680 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_33 2023-12-10T19:20:21.680 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:21.680 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:21.680 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_33 2023-12-10T19:20:21.680 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_33 2023-12-10T19:20:21.779 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_33 2023-12-10T19:20:22.074 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr:2023-12-10T19:20:22.076+0000 7f2c5add4640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5655570e2b60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-10T19:20:22.074 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr:2023-12-10T19:20:22.076+0000 7f2c60de0640 -1 rbd::mirror::ImageReplayer: 0x5655574bcc80 [3/195ff7a6-d9ef-4fb6-afac-e87b92563048] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-10T19:20:23.016 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-10T19:20:23.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:20:23.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_34 2023-12-10T19:20:23.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror image_34 snap 2023-12-10T19:20:23.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:23.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:23.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_34 2023-12-10T19:20:23.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap 2023-12-10T19:20:23.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/image_34@snap 2023-12-10T19:20:23.693 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:20:23.711 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image_retry cluster2 mirror image_34 2023-12-10T19:20:23.711 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:23.711 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:23.711 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_34 2023-12-10T19:20:23.711 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:20:23.712 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_34 2023-12-10T19:20:23.712 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:23.712 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:23.712 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_34 2023-12-10T19:20:23.712 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_34 2023-12-10T19:20:23.822 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_34 2023-12-10T19:20:23.874 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.27184819969357776) > weight (0.25) 2023-12-10T19:20:23.874 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.5604112340320591) > weight (0.25) 2023-12-10T19:20:23.874 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2023-12-10T19:20:23.875 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2023-12-10T19:20:23.875 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2023-12-10T19:20:23.876 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi076.stderr:2023-12-10T19:20:23.880+0000 7f608c1ec640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 53056) UID: 1000 2023-12-10T19:20:23.974 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi076.stderr:2023-12-10T19:20:23.976+0000 7f55882ed640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556b576fcea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-10T19:20:23.974 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi076.stderr:2023-12-10T19:20:23.976+0000 7f558eafa640 -1 rbd::mirror::ImageReplayer: 0x556b564dcf00 [3/d25b8883-1661-4a4c-bc7a-8d186c1f1d26] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-10T19:20:24.863 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-10T19:20:24.868 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:20:24.868 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_35 2023-12-10T19:20:24.868 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror image_35 snap 2023-12-10T19:20:24.868 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:24.868 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:24.868 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_35 2023-12-10T19:20:24.868 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap 2023-12-10T19:20:24.868 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/image_35@snap 2023-12-10T19:20:25.701 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:20:25.729 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image_retry cluster2 mirror image_35 2023-12-10T19:20:25.729 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:25.729 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:25.729 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_35 2023-12-10T19:20:25.729 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:20:25.729 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_35 2023-12-10T19:20:25.729 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:25.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:25.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_35 2023-12-10T19:20:25.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_35 2023-12-10T19:20:25.822 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_35 2023-12-10T19:20:26.084 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi076.stderr:2023-12-10T19:20:26.087+0000 7f7f8e8d6640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562af6554b60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-10T19:20:26.084 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi076.stderr:2023-12-10T19:20:26.087+0000 7f7f950e3640 -1 rbd::mirror::ImageReplayer: 0x562af5485900 [3/fc4fbc1c-ab50-48d4-a59d-0c60a31269a4] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-10T19:20:26.737 INFO:tasks.ceph.cluster2.osd.2.smithi076.stderr:2023-12-10T19:20:26.735+0000 7f7f6add3640 -1 reset not still connected to 0x564308a69040 2023-12-10T19:20:26.737 INFO:tasks.ceph.cluster2.osd.2.smithi076.stderr:2023-12-10T19:20:26.735+0000 7f7f6add3640 -1 reset not still connected to 0x5643145c6680 2023-12-10T19:20:26.737 INFO:tasks.ceph.cluster2.osd.2.smithi076.stderr:2023-12-10T19:20:26.735+0000 7f7f6add3640 -1 reset not still connected to 0x564349c43ee0 2023-12-10T19:20:26.876 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2023-12-10T19:20:26.876 INFO:teuthology.orchestra.run:waiting for 600 2023-12-10T19:20:26.877 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2023-12-10T19:20:26.877 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2023-12-10T19:20:26.877 DEBUG:teuthology.orchestra.run.smithi076:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2023-12-10T19:20:26.879 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2023-12-10T19:20:26.879 INFO:tasks.rbd_mirror_thrash:waiting for 89 secs before thrashing 2023-12-10T19:20:27.085 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-10T19:20:27.089 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:20:27.089 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_36 2023-12-10T19:20:27.089 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror image_36 snap 2023-12-10T19:20:27.089 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:27.089 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:27.089 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_36 2023-12-10T19:20:27.089 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap 2023-12-10T19:20:27.090 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/image_36@snap 2023-12-10T19:20:27.700 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:20:27.709 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image_retry cluster2 mirror image_36 2023-12-10T19:20:27.709 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:27.709 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:27.710 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_36 2023-12-10T19:20:27.710 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:20:27.710 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_36 2023-12-10T19:20:27.710 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:27.710 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:27.710 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_36 2023-12-10T19:20:27.711 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_36 2023-12-10T19:20:27.794 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_36 2023-12-10T19:20:27.988 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi076.stderr:2023-12-10T19:20:27.991+0000 7f7f8e8d6640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562af5b0cd00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-10T19:20:27.988 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi076.stderr:2023-12-10T19:20:27.991+0000 7f7f948e2640 -1 rbd::mirror::ImageReplayer: 0x562af5485680 [3/ad2cd5bc-da17-44b5-9afd-c05bcf6f8746] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-10T19:20:29.014 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-10T19:20:29.017 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:20:29.018 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_37 2023-12-10T19:20:29.018 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror image_37 snap 2023-12-10T19:20:29.018 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:29.018 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:29.018 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_37 2023-12-10T19:20:29.018 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap 2023-12-10T19:20:29.018 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/image_37@snap 2023-12-10T19:20:29.538 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:20:29.572 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image_retry cluster2 mirror image_37 2023-12-10T19:20:29.572 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:29.572 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:29.572 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_37 2023-12-10T19:20:29.573 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:20:29.576 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_37 2023-12-10T19:20:29.576 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:29.576 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:29.576 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_37 2023-12-10T19:20:29.576 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_37 2023-12-10T19:20:29.658 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_37 2023-12-10T19:20:29.830 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi076.stderr:2023-12-10T19:20:29.831+0000 7f55882ed640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556b576fd380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-10T19:20:29.830 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi076.stderr:2023-12-10T19:20:29.831+0000 7f558eafa640 -1 rbd::mirror::ImageReplayer: 0x556b564dc280 [3/a4d8c861-aa4e-4c85-9970-1f362ca1c6b6] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-10T19:20:31.089 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-10T19:20:31.098 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:20:31.098 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_38 2023-12-10T19:20:31.098 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror image_38 snap 2023-12-10T19:20:31.098 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:31.098 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:31.098 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_38 2023-12-10T19:20:31.098 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap 2023-12-10T19:20:31.098 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/image_38@snap 2023-12-10T19:20:31.733 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:20:31.745 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image_retry cluster2 mirror image_38 2023-12-10T19:20:31.746 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:31.746 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:31.746 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_38 2023-12-10T19:20:31.746 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:20:31.747 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_38 2023-12-10T19:20:31.747 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:31.747 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:31.747 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_38 2023-12-10T19:20:31.747 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_38 2023-12-10T19:20:31.829 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_38 2023-12-10T19:20:31.882 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-12-10T19:20:31.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-12-10T19:20:31.882 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2023-12-10T19:20:31.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr:2023-12-10T19:20:31.887+0000 7f3f4f45c640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 61750) UID: 1000 2023-12-10T19:20:32.112 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi076.stderr:2023-12-10T19:20:32.119+0000 7f8341b82640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55bc880189c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-10T19:20:32.112 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi076.stderr:2023-12-10T19:20:32.119+0000 7f8348b90640 -1 rbd::mirror::ImageReplayer: 0x55bc87942280 [3/82e2c879-d2a6-4227-b47b-c52a54933f28] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-10T19:20:33.026 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-10T19:20:33.035 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:20:33.035 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_39 2023-12-10T19:20:33.035 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror image_39 snap 2023-12-10T19:20:33.035 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:33.035 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:33.036 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_39 2023-12-10T19:20:33.036 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap 2023-12-10T19:20:33.036 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/image_39@snap 2023-12-10T19:20:33.755 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:20:33.779 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image_retry cluster2 mirror image_39 2023-12-10T19:20:33.779 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:33.779 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:33.780 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_39 2023-12-10T19:20:33.780 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:20:33.780 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_39 2023-12-10T19:20:33.780 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:33.780 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:33.780 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_39 2023-12-10T19:20:33.780 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_39 2023-12-10T19:20:33.883 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-12-10T19:20:33.883 INFO:teuthology.orchestra.run:waiting for 600 2023-12-10T19:20:33.883 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-12-10T19:20:33.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-10T19:20:33.884 DEBUG:teuthology.orchestra.run.smithi076:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-10T19:20:33.884 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_39 2023-12-10T19:20:33.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-10T19:20:33.886 INFO:tasks.rbd_mirror_thrash:waiting for 66 secs before thrashing 2023-12-10T19:20:34.042 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi076.stderr:2023-12-10T19:20:34.047+0000 7f55882ed640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556b56c65d40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-10T19:20:34.042 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi076.stderr:2023-12-10T19:20:34.047+0000 7f558eafa640 -1 rbd::mirror::ImageReplayer: 0x556b57534a00 [3/b649fd8b-2f2a-4f7d-94c5-3e9e6a6f4c99] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-10T19:20:34.825 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-10T19:20:34.840 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:20:34.840 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_40 2023-12-10T19:20:34.840 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror image_40 snap 2023-12-10T19:20:34.840 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:34.841 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:34.841 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_40 2023-12-10T19:20:34.841 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap 2023-12-10T19:20:34.841 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/image_40@snap 2023-12-10T19:20:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:20:35.786 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image_retry cluster2 mirror image_40 2023-12-10T19:20:35.786 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:35.786 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:35.786 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_40 2023-12-10T19:20:35.786 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:20:35.787 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_40 2023-12-10T19:20:35.788 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:35.788 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:35.788 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_40 2023-12-10T19:20:35.788 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_40 2023-12-10T19:20:35.897 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_40 2023-12-10T19:20:36.133 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi076.stderr:2023-12-10T19:20:36.139+0000 7f55882ed640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556b56d221a0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-10T19:20:36.133 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi076.stderr:2023-12-10T19:20:36.139+0000 7f558eafa640 -1 rbd::mirror::ImageReplayer: 0x556b564dc500 [3/36d7ae4a-9a61-4f12-8a89-e0b759776b33] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-10T19:20:37.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-10T19:20:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:20:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_41 2023-12-10T19:20:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror image_41 snap 2023-12-10T19:20:37.085 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:37.085 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:37.085 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_41 2023-12-10T19:20:37.085 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap 2023-12-10T19:20:37.085 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/image_41@snap 2023-12-10T19:20:37.798 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:20:37.816 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image_retry cluster2 mirror image_41 2023-12-10T19:20:37.816 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:37.817 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:37.817 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_41 2023-12-10T19:20:37.817 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:20:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_41 2023-12-10T19:20:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_41 2023-12-10T19:20:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_41 2023-12-10T19:20:37.913 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_41 2023-12-10T19:20:38.080 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr:2023-12-10T19:20:38.083+0000 7f2c5a5d3640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5655570e29c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-10T19:20:38.080 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr:2023-12-10T19:20:38.083+0000 7f2c615e1640 -1 rbd::mirror::ImageReplayer: 0x5655574bc780 [3/0caf4caf-9ee1-436f-874d-71291f8008fb] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-10T19:20:39.053 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-10T19:20:39.057 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:20:39.058 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_42 2023-12-10T19:20:39.058 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror image_42 snap 2023-12-10T19:20:39.058 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:39.058 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:39.058 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_42 2023-12-10T19:20:39.058 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap 2023-12-10T19:20:39.058 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/image_42@snap 2023-12-10T19:20:39.537 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:20:39.545 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image_retry cluster2 mirror image_42 2023-12-10T19:20:39.546 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:39.546 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:39.546 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_42 2023-12-10T19:20:39.546 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:20:39.546 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_42 2023-12-10T19:20:39.546 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:39.546 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:39.546 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_42 2023-12-10T19:20:39.546 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_42 2023-12-10T19:20:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_42 2023-12-10T19:20:39.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi076.stderr:2023-12-10T19:20:39.767+0000 7f3eb9254640 -1 rbd::mirror::ImageMap: 0x55b33c05a360 handle_peer_ack_remove: failed to remove global_image_id=472ccd57-dfb6-4e9c-bb2d-657bb53c8829 2023-12-10T19:20:40.026 INFO:tasks.ceph.cluster1.osd.2.smithi033.stderr:2023-12-10T19:20:40.014+0000 7fef8ab8e640 -1 reset not still connected to 0x55c5a1a855f0 2023-12-10T19:20:40.026 INFO:tasks.ceph.cluster1.osd.2.smithi033.stderr:2023-12-10T19:20:40.014+0000 7fef8ab8e640 -1 reset not still connected to 0x55c5a1a85790 2023-12-10T19:20:40.026 INFO:tasks.ceph.cluster1.osd.2.smithi033.stderr:2023-12-10T19:20:40.014+0000 7fef8ab8e640 -1 reset not still connected to 0x55c5a20ef380 2023-12-10T19:20:40.026 INFO:tasks.ceph.cluster1.osd.2.smithi033.stderr:2023-12-10T19:20:40.014+0000 7fef8ab8e640 -1 reset not still connected to 0x55c5a385b5f0 2023-12-10T19:20:41.072 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-10T19:20:41.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:20:41.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_43 2023-12-10T19:20:41.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror image_43 snap 2023-12-10T19:20:41.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:41.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:41.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_43 2023-12-10T19:20:41.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap 2023-12-10T19:20:41.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/image_43@snap 2023-12-10T19:20:41.830 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:20:41.850 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image_retry cluster2 mirror image_43 2023-12-10T19:20:41.850 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:41.850 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:41.850 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_43 2023-12-10T19:20:41.850 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:20:41.852 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_43 2023-12-10T19:20:41.852 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:41.852 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:41.852 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_43 2023-12-10T19:20:41.852 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_43 2023-12-10T19:20:41.944 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_43 2023-12-10T19:20:42.108 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi076.stderr:2023-12-10T19:20:42.115+0000 7f7f8e0d5640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562af6554ea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-10T19:20:42.108 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi076.stderr:2023-12-10T19:20:42.115+0000 7f7f950e3640 -1 rbd::mirror::ImageReplayer: 0x562af5e16000 [3/bc83a314-8e55-4363-aaa1-1ab49240de8f] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-10T19:20:43.188 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-10T19:20:43.192 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:20:43.192 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_44 2023-12-10T19:20:43.192 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror image_44 snap 2023-12-10T19:20:43.192 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:43.192 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:43.192 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_44 2023-12-10T19:20:43.192 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap 2023-12-10T19:20:43.192 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/image_44@snap 2023-12-10T19:20:43.868 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:20:43.911 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image_retry cluster2 mirror image_44 2023-12-10T19:20:43.911 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:43.911 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:43.911 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_44 2023-12-10T19:20:43.911 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:20:43.912 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_44 2023-12-10T19:20:43.912 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:43.912 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:43.912 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_44 2023-12-10T19:20:43.912 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_44 2023-12-10T19:20:43.986 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_44 2023-12-10T19:20:44.377 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi076.stderr:2023-12-10T19:20:44.382+0000 7f558f2fb640 -1 rbd::mirror::ImageMap: 0x556b5732e120 handle_peer_ack_remove: failed to remove global_image_id=e95c6ade-d549-422b-a9cd-187a5dd6a0d6 2023-12-10T19:20:44.900 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-10T19:20:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:20:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_45 2023-12-10T19:20:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror image_45 snap 2023-12-10T19:20:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_45 2023-12-10T19:20:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap 2023-12-10T19:20:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/image_45@snap 2023-12-10T19:20:45.855 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:20:45.874 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image_retry cluster2 mirror image_45 2023-12-10T19:20:45.874 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:45.874 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:45.874 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_45 2023-12-10T19:20:45.874 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:20:45.874 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_45 2023-12-10T19:20:45.875 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:45.875 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:45.875 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_45 2023-12-10T19:20:45.875 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_45 2023-12-10T19:20:45.970 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_45 2023-12-10T19:20:46.203 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr:2023-12-10T19:20:46.206+0000 7f2c5a5d3640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56555717e9c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-10T19:20:46.204 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr:2023-12-10T19:20:46.206+0000 7f2c60de0640 -1 rbd::mirror::ImageReplayer: 0x5655574bd680 [3/40a91520-4e7a-4ba5-aaf0-64312a2dcb5a] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-10T19:20:47.161 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-10T19:20:47.168 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:20:47.168 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_46 2023-12-10T19:20:47.168 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror image_46 snap 2023-12-10T19:20:47.168 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:47.168 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:47.168 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_46 2023-12-10T19:20:47.169 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap 2023-12-10T19:20:47.169 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/image_46@snap 2023-12-10T19:20:47.865 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:20:47.878 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image_retry cluster2 mirror image_46 2023-12-10T19:20:47.878 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:47.878 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:47.878 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_46 2023-12-10T19:20:47.878 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:20:47.879 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_46 2023-12-10T19:20:47.879 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:47.879 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:47.879 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_46 2023-12-10T19:20:47.879 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_46 2023-12-10T19:20:48.018 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_46 2023-12-10T19:20:48.404 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr:2023-12-10T19:20:48.410+0000 7f2c5add4640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5655570e3520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-10T19:20:48.404 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr:2023-12-10T19:20:48.410+0000 7f2c615e1640 -1 rbd::mirror::ImageReplayer: 0x565556ff0500 [3/509368f6-035e-4be6-926a-dc4bce662e88] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-10T19:20:49.244 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-10T19:20:49.248 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:20:49.249 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_47 2023-12-10T19:20:49.249 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror image_47 snap 2023-12-10T19:20:49.249 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:49.249 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:49.249 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_47 2023-12-10T19:20:49.249 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap 2023-12-10T19:20:49.249 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/image_47@snap 2023-12-10T19:20:49.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:20:49.565 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image_retry cluster2 mirror image_47 2023-12-10T19:20:49.565 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:49.565 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:49.566 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_47 2023-12-10T19:20:49.566 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:20:49.566 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_47 2023-12-10T19:20:49.566 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:49.566 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:49.566 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_47 2023-12-10T19:20:49.566 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_47 2023-12-10T19:20:49.658 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_47 2023-12-10T19:20:49.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi076.stderr:2023-12-10T19:20:49.806+0000 7f3eb8a53640 -1 rbd::mirror::ImageMap: 0x55b33c05a360 handle_peer_ack_remove: failed to remove global_image_id=35463a96-219e-457b-aaa2-28a56599112e 2023-12-10T19:20:50.853 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-10T19:20:50.857 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:20:50.858 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_48 2023-12-10T19:20:50.858 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror image_48 snap 2023-12-10T19:20:50.858 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:50.858 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:50.858 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_48 2023-12-10T19:20:50.858 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap 2023-12-10T19:20:50.858 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/image_48@snap 2023-12-10T19:20:51.556 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:20:51.573 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image_retry cluster2 mirror image_48 2023-12-10T19:20:51.573 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:51.573 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:51.574 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_48 2023-12-10T19:20:51.574 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:20:51.574 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_48 2023-12-10T19:20:51.574 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:51.574 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:51.574 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_48 2023-12-10T19:20:51.574 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_48 2023-12-10T19:20:51.664 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_48 2023-12-10T19:20:51.859 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi076.stderr:2023-12-10T19:20:51.862+0000 7f7f8e0d5640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562af6555860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-10T19:20:51.859 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi076.stderr:2023-12-10T19:20:51.862+0000 7f7f950e3640 -1 rbd::mirror::ImageReplayer: 0x562af5e16f00 [3/935218e9-f0e7-416b-a035-66ab9afc74f5] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-10T19:20:52.876 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-10T19:20:52.881 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:20:52.881 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_49 2023-12-10T19:20:52.881 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror image_49 snap 2023-12-10T19:20:52.881 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:52.881 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:52.881 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_49 2023-12-10T19:20:52.882 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap 2023-12-10T19:20:52.882 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/image_49@snap 2023-12-10T19:20:53.567 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:20:53.588 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image_retry cluster2 mirror image_49 2023-12-10T19:20:53.589 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:53.589 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:53.589 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_49 2023-12-10T19:20:53.589 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:20:53.589 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_49 2023-12-10T19:20:53.589 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:53.589 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:53.589 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_49 2023-12-10T19:20:53.590 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_49 2023-12-10T19:20:53.711 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_49 2023-12-10T19:20:54.923 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-10T19:20:54.926 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:20:54.927 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_50 2023-12-10T19:20:54.927 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_snapshot cluster2 mirror image_50 snap 2023-12-10T19:20:54.927 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:54.927 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:54.927 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_50 2023-12-10T19:20:54.927 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local snap=snap 2023-12-10T19:20:54.927 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap rm mirror/image_50@snap 2023-12-10T19:20:55.575 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing snap: 100% complete...done. 2023-12-10T19:20:55.592 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image_retry cluster2 mirror image_50 2023-12-10T19:20:55.592 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:55.593 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:55.593 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_50 2023-12-10T19:20:55.593 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:20:55.608 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_50 2023-12-10T19:20:55.608 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:20:55.608 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:55.608 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_50 2023-12-10T19:20:55.608 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_50 2023-12-10T19:20:55.757 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_50 2023-12-10T19:20:55.918 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi076.stderr:2023-12-10T19:20:55.922+0000 7f5588aee640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556b574ff380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-10T19:20:55.918 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi076.stderr:2023-12-10T19:20:55.922+0000 7f558eafa640 -1 rbd::mirror::ImageReplayer: 0x556b564dcf00 [3/8d5c864a-1785-4fbb-8bed-7f9006bb0d2b] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-10T19:20:56.913 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-10T19:20:56.924 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:20:56.924 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ testlog TEST: image deletions should propagate 2023-12-10T19:20:56.924 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ tee -a /tmp/tmp.qNjRyMq75D/rbd-mirror.test.log 2023-12-10T19:20:56.934 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ date +%F %T 2023-12-10T19:20:56.935 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo 2023-12-10 19:20:56 TEST: image deletions should propagate 2023-12-10T19:20:56.936 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:2023-12-10 19:20:56 TEST: image deletions should propagate 2023-12-10T19:20:56.936 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_pool_images cluster1 mirror 0 2023-12-10T19:20:56.936 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:20:56.936 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:20:56.936 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_count=0 2023-12-10T19:20:56.936 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:20:56.936 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local count 2023-12-10T19:20:56.936 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local last_count=0 2023-12-10T19:20:56.936 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:20:56.937 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 1 40 2023-12-10T19:20:56.937 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 1 -ne 1 2023-12-10T19:20:56.937 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ + rbd --cluster cluster1 -p mirror mirror pool status 2023-12-10T19:20:56.937 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:cut -d -f 2 2023-12-10T19:20:56.956 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep images: 2023-12-10T19:20:56.995 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: failed to query services: (13) Permission denied 2023-12-10T19:20:56.999 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ count=2 2023-12-10T19:20:56.999 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 2 = 0 2023-12-10T19:20:56.999 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 2 -ne 0 2023-12-10T19:20:56.999 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ break 2023-12-10T19:20:56.999 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 2 -eq 0 2023-12-10T19:20:56.999 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ last_count=2 2023-12-10T19:20:56.999 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ true 2023-12-10T19:20:56.999 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 1 40 2023-12-10T19:20:57.000 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 1 -ne 1 2023-12-10T19:20:57.000 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror mirror pool status 2023-12-10T19:20:57.001 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep images: 2023-12-10T19:20:57.001 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cut -d -f 2 2023-12-10T19:20:57.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: failed to query services: (13) Permission denied 2023-12-10T19:20:57.080 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ count=2 2023-12-10T19:20:57.080 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 2 = 0 2023-12-10T19:20:57.080 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 2 -ne 2 2023-12-10T19:20:57.080 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 2 -ne 1 2023-12-10T19:20:57.080 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 30 2023-12-10T19:21:27.083 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror mirror pool status 2023-12-10T19:21:27.083 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep images: 2023-12-10T19:21:27.083 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cut -d -f 2 2023-12-10T19:21:27.132 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: failed to query services: (13) Permission denied 2023-12-10T19:21:27.137 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ count=0 2023-12-10T19:21:27.137 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 0 = 0 2023-12-10T19:21:27.138 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:21:27.138 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_pool_healthy cluster1 mirror 0 2023-12-10T19:21:27.138 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:27.138 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:27.138 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:21:27.138 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state 2023-12-10T19:21:27.138 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 1 40 2023-12-10T19:21:27.138 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test 1 -ne 1 2023-12-10T19:21:27.139 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror mirror pool status 2023-12-10T19:21:27.139 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ grep image health: 2023-12-10T19:21:27.139 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cut -d -f 3 2023-12-10T19:21:27.180 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: failed to query services: (13) Permission denied 2023-12-10T19:21:27.183 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ state=OK 2023-12-10T19:21:27.183 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test OK = ERROR 2023-12-10T19:21:27.183 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test OK = OK 2023-12-10T19:21:27.183 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:21:27.183 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 1 50 2023-12-10T19:21:27.185 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_1 2023-12-10T19:21:27.185 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2023-12-10T19:21:27.185 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:27.185 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:27.185 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_1 2023-12-10T19:21:27.185 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=deleted 2023-12-10T19:21:27.185 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:27.185 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:21:27.185 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:27.185 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_1 2023-12-10T19:21:27.186 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:27.186 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:27.186 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_1 2023-12-10T19:21:27.186 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_1 2023-12-10T19:21:27.186 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:27.222 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_1: (2) No such file or directory 2023-12-10T19:21:27.225 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image_id= 2023-12-10T19:21:27.225 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.1 2023-12-10T19:21:27.326 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_1 deleted 2023-12-10T19:21:27.326 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:27.327 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:27.327 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_1 2023-12-10T19:21:27.327 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=deleted 2023-12-10T19:21:27.327 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:27.327 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:21:27.327 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:21:27.327 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_1 2023-12-10T19:21:27.327 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:27.327 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:27.327 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_1 2023-12-10T19:21:27.328 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_1 2023-12-10T19:21:27.328 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:27.369 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_1: (2) No such file or directory 2023-12-10T19:21:27.373 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:21:27.373 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:27.373 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test deleted = deleted 2023-12-10T19:21:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:21:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_2 2023-12-10T19:21:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2023-12-10T19:21:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_2 2023-12-10T19:21:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=deleted 2023-12-10T19:21:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:21:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_2 2023-12-10T19:21:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_2 2023-12-10T19:21:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_2 2023-12-10T19:21:27.375 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_2: (2) No such file or directory 2023-12-10T19:21:27.414 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image_id= 2023-12-10T19:21:27.414 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.1 2023-12-10T19:21:27.516 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_2 deleted 2023-12-10T19:21:27.517 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:27.517 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:27.517 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_2 2023-12-10T19:21:27.517 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=deleted 2023-12-10T19:21:27.517 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:27.517 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:21:27.517 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:21:27.517 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_2 2023-12-10T19:21:27.517 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:27.517 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:27.517 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_2 2023-12-10T19:21:27.517 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_2 2023-12-10T19:21:27.517 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:27.564 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_2: (2) No such file or directory 2023-12-10T19:21:27.568 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:21:27.568 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:27.568 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test deleted = deleted 2023-12-10T19:21:27.568 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:21:27.568 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_3 2023-12-10T19:21:27.568 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2023-12-10T19:21:27.568 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:27.569 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:27.569 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_3 2023-12-10T19:21:27.569 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=deleted 2023-12-10T19:21:27.569 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:27.569 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:21:27.569 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:27.569 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_3 2023-12-10T19:21:27.569 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:27.569 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:27.569 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_3 2023-12-10T19:21:27.569 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_3 2023-12-10T19:21:27.569 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:27.606 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_3: (2) No such file or directory 2023-12-10T19:21:27.610 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image_id= 2023-12-10T19:21:27.610 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.1 2023-12-10T19:21:27.712 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_3 deleted 2023-12-10T19:21:27.712 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:27.712 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:27.712 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_3 2023-12-10T19:21:27.712 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=deleted 2023-12-10T19:21:27.712 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:27.712 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:21:27.712 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:21:27.712 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_3 2023-12-10T19:21:27.712 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:27.712 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:27.712 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_3 2023-12-10T19:21:27.712 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_3 2023-12-10T19:21:27.712 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:27.758 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_3: (2) No such file or directory 2023-12-10T19:21:27.762 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:21:27.762 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:27.762 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test deleted = deleted 2023-12-10T19:21:27.762 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:21:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_4 2023-12-10T19:21:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2023-12-10T19:21:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_4 2023-12-10T19:21:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=deleted 2023-12-10T19:21:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:21:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_4 2023-12-10T19:21:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_4 2023-12-10T19:21:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_4 2023-12-10T19:21:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:27.801 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_4: (2) No such file or directory 2023-12-10T19:21:27.805 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image_id= 2023-12-10T19:21:27.805 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.1 2023-12-10T19:21:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_4 deleted 2023-12-10T19:21:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_4 2023-12-10T19:21:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=deleted 2023-12-10T19:21:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:21:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:21:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_4 2023-12-10T19:21:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:27.908 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_4 2023-12-10T19:21:27.908 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_4 2023-12-10T19:21:27.908 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:27.956 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_4: (2) No such file or directory 2023-12-10T19:21:27.960 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:21:27.960 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:27.960 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test deleted = deleted 2023-12-10T19:21:27.960 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:21:27.961 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_5 2023-12-10T19:21:27.961 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2023-12-10T19:21:27.961 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:27.961 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:27.961 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_5 2023-12-10T19:21:27.961 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=deleted 2023-12-10T19:21:27.961 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:27.961 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:21:27.961 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:27.961 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_5 2023-12-10T19:21:27.961 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:27.961 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:27.961 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_5 2023-12-10T19:21:27.961 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_5 2023-12-10T19:21:27.961 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:27.999 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_5: (2) No such file or directory 2023-12-10T19:21:28.004 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image_id= 2023-12-10T19:21:28.004 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.1 2023-12-10T19:21:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_5 deleted 2023-12-10T19:21:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_5 2023-12-10T19:21:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=deleted 2023-12-10T19:21:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:21:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:21:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_5 2023-12-10T19:21:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_5 2023-12-10T19:21:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_5 2023-12-10T19:21:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:28.149 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_5: (2) No such file or directory 2023-12-10T19:21:28.153 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:21:28.154 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:28.154 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test deleted = deleted 2023-12-10T19:21:28.154 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:21:28.154 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_6 2023-12-10T19:21:28.154 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2023-12-10T19:21:28.154 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:28.154 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:28.154 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_6 2023-12-10T19:21:28.154 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=deleted 2023-12-10T19:21:28.154 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:28.154 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:21:28.154 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:28.154 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_6 2023-12-10T19:21:28.154 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:28.154 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:28.155 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_6 2023-12-10T19:21:28.155 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_6 2023-12-10T19:21:28.155 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:28.191 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_6: (2) No such file or directory 2023-12-10T19:21:28.195 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image_id= 2023-12-10T19:21:28.195 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.1 2023-12-10T19:21:28.298 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_6 deleted 2023-12-10T19:21:28.298 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:28.298 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:28.298 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_6 2023-12-10T19:21:28.298 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=deleted 2023-12-10T19:21:28.298 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:28.298 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:21:28.298 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:21:28.298 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_6 2023-12-10T19:21:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_6 2023-12-10T19:21:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_6 2023-12-10T19:21:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:28.343 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_6: (2) No such file or directory 2023-12-10T19:21:28.348 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:21:28.348 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:28.348 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test deleted = deleted 2023-12-10T19:21:28.348 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:21:28.348 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_7 2023-12-10T19:21:28.348 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2023-12-10T19:21:28.348 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:28.348 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:28.348 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_7 2023-12-10T19:21:28.348 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=deleted 2023-12-10T19:21:28.348 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:28.349 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:21:28.349 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:28.349 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_7 2023-12-10T19:21:28.349 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:28.349 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:28.349 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_7 2023-12-10T19:21:28.349 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_7 2023-12-10T19:21:28.349 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:28.385 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_7: (2) No such file or directory 2023-12-10T19:21:28.389 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image_id= 2023-12-10T19:21:28.389 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.1 2023-12-10T19:21:28.491 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_7 deleted 2023-12-10T19:21:28.491 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:28.491 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:28.491 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_7 2023-12-10T19:21:28.491 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=deleted 2023-12-10T19:21:28.491 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:28.492 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:21:28.492 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:21:28.492 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_7 2023-12-10T19:21:28.492 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:28.492 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:28.492 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_7 2023-12-10T19:21:28.492 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_7 2023-12-10T19:21:28.492 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:28.535 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_7: (2) No such file or directory 2023-12-10T19:21:28.540 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:21:28.540 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:28.540 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test deleted = deleted 2023-12-10T19:21:28.540 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:21:28.540 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_8 2023-12-10T19:21:28.540 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2023-12-10T19:21:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_8 2023-12-10T19:21:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=deleted 2023-12-10T19:21:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:21:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_8 2023-12-10T19:21:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_8 2023-12-10T19:21:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_8 2023-12-10T19:21:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:28.578 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_8: (2) No such file or directory 2023-12-10T19:21:28.581 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image_id= 2023-12-10T19:21:28.582 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.1 2023-12-10T19:21:28.684 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_8 deleted 2023-12-10T19:21:28.684 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:28.684 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:28.684 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_8 2023-12-10T19:21:28.684 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=deleted 2023-12-10T19:21:28.684 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:28.684 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:21:28.684 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:21:28.685 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_8 2023-12-10T19:21:28.685 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:28.685 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:28.685 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_8 2023-12-10T19:21:28.685 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_8 2023-12-10T19:21:28.685 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:28.728 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_8: (2) No such file or directory 2023-12-10T19:21:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:21:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test deleted = deleted 2023-12-10T19:21:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:21:28.733 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_9 2023-12-10T19:21:28.733 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_present cluster1 mirror image_9 deleted 2023-12-10T19:21:28.733 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:28.733 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:28.733 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_9 2023-12-10T19:21:28.733 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=deleted 2023-12-10T19:21:28.733 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:28.733 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:21:28.733 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:28.733 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_9 2023-12-10T19:21:28.733 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:28.733 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:28.733 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_9 2023-12-10T19:21:28.733 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_9 2023-12-10T19:21:28.733 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:28.771 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_9: (2) No such file or directory 2023-12-10T19:21:28.774 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image_id= 2023-12-10T19:21:28.774 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.1 2023-12-10T19:21:28.876 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_9 deleted 2023-12-10T19:21:28.876 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:28.876 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:28.876 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_9 2023-12-10T19:21:28.876 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=deleted 2023-12-10T19:21:28.876 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:28.876 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:21:28.876 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:21:28.876 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_9 2023-12-10T19:21:28.876 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:28.876 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:28.877 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_9 2023-12-10T19:21:28.877 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_9 2023-12-10T19:21:28.877 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:28.922 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_9: (2) No such file or directory 2023-12-10T19:21:28.926 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:21:28.926 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:28.926 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test deleted = deleted 2023-12-10T19:21:28.926 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:21:28.927 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_10 2023-12-10T19:21:28.927 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_present cluster1 mirror image_10 deleted 2023-12-10T19:21:28.927 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:28.927 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:28.927 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_10 2023-12-10T19:21:28.927 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=deleted 2023-12-10T19:21:28.927 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:28.928 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:21:28.928 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:28.928 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_10 2023-12-10T19:21:28.928 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:28.928 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:28.928 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_10 2023-12-10T19:21:28.928 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_10 2023-12-10T19:21:28.928 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:28.966 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_10: (2) No such file or directory 2023-12-10T19:21:28.970 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image_id= 2023-12-10T19:21:28.970 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.1 2023-12-10T19:21:29.072 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_10 deleted 2023-12-10T19:21:29.072 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:29.072 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:29.072 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_10 2023-12-10T19:21:29.072 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=deleted 2023-12-10T19:21:29.072 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:29.072 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:21:29.072 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:21:29.072 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_10 2023-12-10T19:21:29.072 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:29.072 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:29.072 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_10 2023-12-10T19:21:29.072 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_10 2023-12-10T19:21:29.073 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:29.119 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_10: (2) No such file or directory 2023-12-10T19:21:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:21:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test deleted = deleted 2023-12-10T19:21:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:21:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_11 2023-12-10T19:21:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_present cluster1 mirror image_11 deleted 2023-12-10T19:21:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_11 2023-12-10T19:21:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=deleted 2023-12-10T19:21:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:21:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_11 2023-12-10T19:21:29.125 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:29.125 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:29.125 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_11 2023-12-10T19:21:29.125 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_11 2023-12-10T19:21:29.125 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:29.162 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_11: (2) No such file or directory 2023-12-10T19:21:29.166 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image_id= 2023-12-10T19:21:29.166 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.1 2023-12-10T19:21:29.268 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_11 deleted 2023-12-10T19:21:29.268 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:29.268 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:29.268 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_11 2023-12-10T19:21:29.268 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=deleted 2023-12-10T19:21:29.268 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:29.268 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:21:29.268 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:21:29.268 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_11 2023-12-10T19:21:29.268 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:29.268 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:29.269 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_11 2023-12-10T19:21:29.269 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_11 2023-12-10T19:21:29.269 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:29.315 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_11: (2) No such file or directory 2023-12-10T19:21:29.321 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:21:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test deleted = deleted 2023-12-10T19:21:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:21:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_12 2023-12-10T19:21:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_present cluster1 mirror image_12 deleted 2023-12-10T19:21:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_12 2023-12-10T19:21:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=deleted 2023-12-10T19:21:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:21:29.323 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:29.323 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_12 2023-12-10T19:21:29.323 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:29.323 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:29.323 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_12 2023-12-10T19:21:29.323 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_12 2023-12-10T19:21:29.323 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:29.358 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_12: (2) No such file or directory 2023-12-10T19:21:29.362 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image_id= 2023-12-10T19:21:29.362 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.1 2023-12-10T19:21:29.464 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_12 deleted 2023-12-10T19:21:29.464 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:29.464 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:29.464 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_12 2023-12-10T19:21:29.464 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=deleted 2023-12-10T19:21:29.464 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:29.465 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:21:29.465 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:21:29.465 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_12 2023-12-10T19:21:29.465 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:29.465 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:29.465 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_12 2023-12-10T19:21:29.465 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_12 2023-12-10T19:21:29.465 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:29.510 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_12: (2) No such file or directory 2023-12-10T19:21:29.514 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:21:29.515 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:29.515 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test deleted = deleted 2023-12-10T19:21:29.515 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:21:29.515 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_13 2023-12-10T19:21:29.515 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_present cluster1 mirror image_13 deleted 2023-12-10T19:21:29.515 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:29.515 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:29.515 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_13 2023-12-10T19:21:29.515 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=deleted 2023-12-10T19:21:29.515 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:29.515 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:21:29.515 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:29.515 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_13 2023-12-10T19:21:29.515 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:29.516 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:29.516 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_13 2023-12-10T19:21:29.516 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_13 2023-12-10T19:21:29.516 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:29.553 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_13: (2) No such file or directory 2023-12-10T19:21:29.556 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image_id= 2023-12-10T19:21:29.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.1 2023-12-10T19:21:29.659 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_13 deleted 2023-12-10T19:21:29.659 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:29.659 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:29.659 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_13 2023-12-10T19:21:29.659 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=deleted 2023-12-10T19:21:29.659 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:29.659 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:21:29.659 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:21:29.659 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_13 2023-12-10T19:21:29.659 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:29.659 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:29.659 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_13 2023-12-10T19:21:29.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_13 2023-12-10T19:21:29.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:29.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_13: (2) No such file or directory 2023-12-10T19:21:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:21:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test deleted = deleted 2023-12-10T19:21:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:21:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_14 2023-12-10T19:21:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_present cluster1 mirror image_14 deleted 2023-12-10T19:21:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_14 2023-12-10T19:21:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=deleted 2023-12-10T19:21:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:21:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_14 2023-12-10T19:21:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_14 2023-12-10T19:21:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_14 2023-12-10T19:21:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:29.746 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_14: (2) No such file or directory 2023-12-10T19:21:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image_id= 2023-12-10T19:21:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.1 2023-12-10T19:21:29.851 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_14 deleted 2023-12-10T19:21:29.851 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:29.852 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:29.852 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_14 2023-12-10T19:21:29.852 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=deleted 2023-12-10T19:21:29.852 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:29.852 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:21:29.852 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:21:29.852 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_14 2023-12-10T19:21:29.852 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:29.852 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:29.852 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_14 2023-12-10T19:21:29.852 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_14 2023-12-10T19:21:29.852 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_14: (2) No such file or directory 2023-12-10T19:21:29.900 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:21:29.900 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:29.900 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test deleted = deleted 2023-12-10T19:21:29.901 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:21:29.901 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_15 2023-12-10T19:21:29.901 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_present cluster1 mirror image_15 deleted 2023-12-10T19:21:29.901 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:29.901 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:29.901 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_15 2023-12-10T19:21:29.901 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=deleted 2023-12-10T19:21:29.901 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:29.901 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:21:29.901 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:29.901 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_15 2023-12-10T19:21:29.901 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:29.901 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:29.901 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_15 2023-12-10T19:21:29.901 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_15 2023-12-10T19:21:29.901 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_15: (2) No such file or directory 2023-12-10T19:21:29.942 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image_id= 2023-12-10T19:21:29.942 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.1 2023-12-10T19:21:30.044 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_15 deleted 2023-12-10T19:21:30.044 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_15 2023-12-10T19:21:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=deleted 2023-12-10T19:21:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:21:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:21:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_15 2023-12-10T19:21:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_15 2023-12-10T19:21:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_15 2023-12-10T19:21:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:30.090 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_15: (2) No such file or directory 2023-12-10T19:21:30.095 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:21:30.095 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:30.095 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test deleted = deleted 2023-12-10T19:21:30.095 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:21:30.095 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_16 2023-12-10T19:21:30.095 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_present cluster1 mirror image_16 deleted 2023-12-10T19:21:30.095 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:30.095 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:30.095 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_16 2023-12-10T19:21:30.095 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=deleted 2023-12-10T19:21:30.095 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:30.095 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:21:30.095 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:30.095 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_16 2023-12-10T19:21:30.095 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:30.096 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:30.096 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_16 2023-12-10T19:21:30.096 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_16 2023-12-10T19:21:30.096 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:30.132 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_16: (2) No such file or directory 2023-12-10T19:21:30.135 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image_id= 2023-12-10T19:21:30.136 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.1 2023-12-10T19:21:30.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_16 deleted 2023-12-10T19:21:30.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:30.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:30.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_16 2023-12-10T19:21:30.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=deleted 2023-12-10T19:21:30.238 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:30.238 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:21:30.238 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:21:30.238 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_16 2023-12-10T19:21:30.238 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:30.238 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:30.238 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_16 2023-12-10T19:21:30.238 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_16 2023-12-10T19:21:30.238 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:30.283 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_16: (2) No such file or directory 2023-12-10T19:21:30.287 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:21:30.288 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:30.288 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test deleted = deleted 2023-12-10T19:21:30.288 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:21:30.288 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_17 2023-12-10T19:21:30.288 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_present cluster1 mirror image_17 deleted 2023-12-10T19:21:30.288 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:30.288 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:30.288 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_17 2023-12-10T19:21:30.288 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=deleted 2023-12-10T19:21:30.288 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:30.288 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:21:30.288 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:30.288 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_17 2023-12-10T19:21:30.288 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:30.288 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:30.288 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_17 2023-12-10T19:21:30.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_17 2023-12-10T19:21:30.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:30.325 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_17: (2) No such file or directory 2023-12-10T19:21:30.328 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image_id= 2023-12-10T19:21:30.328 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.1 2023-12-10T19:21:30.430 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_17 deleted 2023-12-10T19:21:30.430 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:30.430 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:30.431 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_17 2023-12-10T19:21:30.431 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=deleted 2023-12-10T19:21:30.431 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:30.431 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:21:30.431 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:21:30.431 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_17 2023-12-10T19:21:30.431 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:30.431 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:30.431 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_17 2023-12-10T19:21:30.431 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_17 2023-12-10T19:21:30.432 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:30.474 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_17: (2) No such file or directory 2023-12-10T19:21:30.479 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:21:30.480 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:30.480 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test deleted = deleted 2023-12-10T19:21:30.480 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:21:30.480 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_18 2023-12-10T19:21:30.480 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_present cluster1 mirror image_18 deleted 2023-12-10T19:21:30.480 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:30.480 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:30.481 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_18 2023-12-10T19:21:30.481 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=deleted 2023-12-10T19:21:30.481 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:30.481 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:21:30.481 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:30.482 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_18 2023-12-10T19:21:30.482 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:30.482 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:30.482 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_18 2023-12-10T19:21:30.482 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_18 2023-12-10T19:21:30.482 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:30.518 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_18: (2) No such file or directory 2023-12-10T19:21:30.522 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image_id= 2023-12-10T19:21:30.522 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.1 2023-12-10T19:21:30.624 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_18 deleted 2023-12-10T19:21:30.624 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:30.624 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:30.624 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_18 2023-12-10T19:21:30.624 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=deleted 2023-12-10T19:21:30.624 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:30.624 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:21:30.624 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:21:30.624 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_18 2023-12-10T19:21:30.624 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:30.625 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:30.625 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_18 2023-12-10T19:21:30.625 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_18 2023-12-10T19:21:30.625 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:30.670 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_18: (2) No such file or directory 2023-12-10T19:21:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:21:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test deleted = deleted 2023-12-10T19:21:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:21:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_19 2023-12-10T19:21:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_present cluster1 mirror image_19 deleted 2023-12-10T19:21:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_19 2023-12-10T19:21:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=deleted 2023-12-10T19:21:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:21:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_19 2023-12-10T19:21:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_19 2023-12-10T19:21:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_19 2023-12-10T19:21:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:30.713 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_19: (2) No such file or directory 2023-12-10T19:21:30.717 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image_id= 2023-12-10T19:21:30.717 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.1 2023-12-10T19:21:30.819 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_19 deleted 2023-12-10T19:21:30.819 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:30.819 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:30.819 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_19 2023-12-10T19:21:30.819 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=deleted 2023-12-10T19:21:30.819 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:30.819 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:21:30.820 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:21:30.820 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_19 2023-12-10T19:21:30.820 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:30.820 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:30.820 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_19 2023-12-10T19:21:30.820 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_19 2023-12-10T19:21:30.820 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:30.864 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_19: (2) No such file or directory 2023-12-10T19:21:30.868 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:21:30.868 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:30.868 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test deleted = deleted 2023-12-10T19:21:30.869 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:21:30.869 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_20 2023-12-10T19:21:30.869 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_present cluster1 mirror image_20 deleted 2023-12-10T19:21:30.869 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:30.869 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:30.869 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_20 2023-12-10T19:21:30.869 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=deleted 2023-12-10T19:21:30.869 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:30.869 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:21:30.869 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:30.869 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_20 2023-12-10T19:21:30.869 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:30.869 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:30.869 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_20 2023-12-10T19:21:30.869 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_20 2023-12-10T19:21:30.870 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:30.905 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_20: (2) No such file or directory 2023-12-10T19:21:30.909 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image_id= 2023-12-10T19:21:30.909 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.1 2023-12-10T19:21:31.011 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_20 deleted 2023-12-10T19:21:31.011 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:31.011 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:31.011 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_20 2023-12-10T19:21:31.011 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=deleted 2023-12-10T19:21:31.011 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:31.011 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:21:31.011 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:21:31.011 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_20 2023-12-10T19:21:31.011 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:31.011 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:31.012 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_20 2023-12-10T19:21:31.012 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_20 2023-12-10T19:21:31.012 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:31.056 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_20: (2) No such file or directory 2023-12-10T19:21:31.061 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:21:31.061 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:31.061 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test deleted = deleted 2023-12-10T19:21:31.061 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:21:31.061 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_21 2023-12-10T19:21:31.061 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_present cluster1 mirror image_21 deleted 2023-12-10T19:21:31.061 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:31.061 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_21 2023-12-10T19:21:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=deleted 2023-12-10T19:21:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:21:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_21 2023-12-10T19:21:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_21 2023-12-10T19:21:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_21 2023-12-10T19:21:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:31.100 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_21: (2) No such file or directory 2023-12-10T19:21:31.103 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image_id= 2023-12-10T19:21:31.103 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.1 2023-12-10T19:21:31.205 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_21 deleted 2023-12-10T19:21:31.205 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:31.205 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:31.205 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_21 2023-12-10T19:21:31.206 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=deleted 2023-12-10T19:21:31.206 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:31.206 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:21:31.206 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:21:31.206 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_21 2023-12-10T19:21:31.206 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:31.206 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:31.206 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_21 2023-12-10T19:21:31.206 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_21 2023-12-10T19:21:31.206 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:31.252 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_21: (2) No such file or directory 2023-12-10T19:21:31.257 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:21:31.257 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:31.257 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test deleted = deleted 2023-12-10T19:21:31.257 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:21:31.257 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_22 2023-12-10T19:21:31.257 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_present cluster1 mirror image_22 deleted 2023-12-10T19:21:31.257 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:31.257 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:31.258 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_22 2023-12-10T19:21:31.258 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=deleted 2023-12-10T19:21:31.258 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:31.258 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:21:31.258 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:31.258 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_22 2023-12-10T19:21:31.258 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:31.258 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:31.258 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_22 2023-12-10T19:21:31.258 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_22 2023-12-10T19:21:31.258 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:31.296 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_22: (2) No such file or directory 2023-12-10T19:21:31.300 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image_id= 2023-12-10T19:21:31.300 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.1 2023-12-10T19:21:31.402 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_22 deleted 2023-12-10T19:21:31.402 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:31.402 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:31.402 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_22 2023-12-10T19:21:31.403 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=deleted 2023-12-10T19:21:31.403 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:31.403 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:21:31.403 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:21:31.403 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_22 2023-12-10T19:21:31.403 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:31.403 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:31.403 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_22 2023-12-10T19:21:31.403 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_22 2023-12-10T19:21:31.403 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:31.448 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_22: (2) No such file or directory 2023-12-10T19:21:31.453 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:21:31.453 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:31.453 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test deleted = deleted 2023-12-10T19:21:31.453 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:21:31.453 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_23 2023-12-10T19:21:31.453 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_present cluster1 mirror image_23 deleted 2023-12-10T19:21:31.453 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:31.453 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:31.453 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_23 2023-12-10T19:21:31.454 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=deleted 2023-12-10T19:21:31.454 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:31.454 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:21:31.454 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:31.454 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_23 2023-12-10T19:21:31.454 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:31.454 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:31.454 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_23 2023-12-10T19:21:31.454 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_23 2023-12-10T19:21:31.454 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:31.491 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_23: (2) No such file or directory 2023-12-10T19:21:31.494 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image_id= 2023-12-10T19:21:31.495 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.1 2023-12-10T19:21:31.596 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_23 deleted 2023-12-10T19:21:31.597 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:31.597 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:31.597 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_23 2023-12-10T19:21:31.597 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=deleted 2023-12-10T19:21:31.597 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:31.597 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:21:31.597 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:21:31.597 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_23 2023-12-10T19:21:31.597 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:31.597 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:31.597 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_23 2023-12-10T19:21:31.597 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_23 2023-12-10T19:21:31.597 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:31.643 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_23: (2) No such file or directory 2023-12-10T19:21:31.648 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:21:31.648 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test deleted = deleted 2023-12-10T19:21:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:21:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_24 2023-12-10T19:21:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_present cluster1 mirror image_24 deleted 2023-12-10T19:21:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_24 2023-12-10T19:21:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=deleted 2023-12-10T19:21:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:21:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_24 2023-12-10T19:21:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_24 2023-12-10T19:21:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_24 2023-12-10T19:21:31.650 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:31.686 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_24: (2) No such file or directory 2023-12-10T19:21:31.689 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image_id= 2023-12-10T19:21:31.690 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.1 2023-12-10T19:21:31.791 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_24 deleted 2023-12-10T19:21:31.791 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:31.792 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:31.792 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_24 2023-12-10T19:21:31.792 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=deleted 2023-12-10T19:21:31.792 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:31.792 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:21:31.792 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:21:31.792 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_24 2023-12-10T19:21:31.792 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:31.792 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:31.792 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_24 2023-12-10T19:21:31.792 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_24 2023-12-10T19:21:31.792 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:31.838 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_24: (2) No such file or directory 2023-12-10T19:21:31.843 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:21:31.843 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:31.843 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test deleted = deleted 2023-12-10T19:21:31.843 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:21:31.843 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_25 2023-12-10T19:21:31.843 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_present cluster1 mirror image_25 deleted 2023-12-10T19:21:31.843 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:31.843 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:31.843 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_25 2023-12-10T19:21:31.843 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=deleted 2023-12-10T19:21:31.843 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:31.844 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:21:31.844 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:31.844 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_25 2023-12-10T19:21:31.844 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:31.844 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:31.844 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_25 2023-12-10T19:21:31.844 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_25 2023-12-10T19:21:31.844 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:31.882 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_25: (2) No such file or directory 2023-12-10T19:21:31.885 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image_id= 2023-12-10T19:21:31.886 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.1 2023-12-10T19:21:31.987 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_25 deleted 2023-12-10T19:21:31.987 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:31.987 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:31.987 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_25 2023-12-10T19:21:31.987 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=deleted 2023-12-10T19:21:31.987 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:31.988 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:21:31.988 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:21:31.988 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_25 2023-12-10T19:21:31.988 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:31.988 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:31.988 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_25 2023-12-10T19:21:31.988 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_25 2023-12-10T19:21:31.988 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:32.033 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_25: (2) No such file or directory 2023-12-10T19:21:32.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:21:32.038 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:32.038 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test deleted = deleted 2023-12-10T19:21:32.038 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:21:32.038 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_26 2023-12-10T19:21:32.038 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_present cluster1 mirror image_26 deleted 2023-12-10T19:21:32.038 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:32.038 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:32.038 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_26 2023-12-10T19:21:32.038 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=deleted 2023-12-10T19:21:32.038 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:32.038 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:21:32.038 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:32.038 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_26 2023-12-10T19:21:32.039 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:32.039 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:32.039 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_26 2023-12-10T19:21:32.039 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_26 2023-12-10T19:21:32.039 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:32.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_26: (2) No such file or directory 2023-12-10T19:21:32.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image_id= 2023-12-10T19:21:32.080 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.1 2023-12-10T19:21:32.181 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_26 deleted 2023-12-10T19:21:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_26 2023-12-10T19:21:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=deleted 2023-12-10T19:21:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:21:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:21:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_26 2023-12-10T19:21:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_26 2023-12-10T19:21:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_26 2023-12-10T19:21:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:32.231 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_26: (2) No such file or directory 2023-12-10T19:21:32.235 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:21:32.235 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:32.235 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test deleted = deleted 2023-12-10T19:21:32.235 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:21:32.235 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_27 2023-12-10T19:21:32.235 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_present cluster1 mirror image_27 deleted 2023-12-10T19:21:32.235 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:32.236 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:32.236 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_27 2023-12-10T19:21:32.236 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=deleted 2023-12-10T19:21:32.236 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:32.236 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:21:32.236 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:32.236 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_27 2023-12-10T19:21:32.236 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:32.236 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:32.236 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_27 2023-12-10T19:21:32.236 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_27 2023-12-10T19:21:32.236 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:32.274 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_27: (2) No such file or directory 2023-12-10T19:21:32.278 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image_id= 2023-12-10T19:21:32.278 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.1 2023-12-10T19:21:32.380 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_27 deleted 2023-12-10T19:21:32.380 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:32.380 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:32.380 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_27 2023-12-10T19:21:32.380 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=deleted 2023-12-10T19:21:32.380 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:32.380 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:21:32.380 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:21:32.380 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_27 2023-12-10T19:21:32.381 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:32.381 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:32.381 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_27 2023-12-10T19:21:32.381 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_27 2023-12-10T19:21:32.381 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:32.426 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_27: (2) No such file or directory 2023-12-10T19:21:32.431 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:21:32.431 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:32.431 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test deleted = deleted 2023-12-10T19:21:32.431 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:21:32.431 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_28 2023-12-10T19:21:32.431 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_present cluster1 mirror image_28 deleted 2023-12-10T19:21:32.431 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:32.431 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:32.431 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_28 2023-12-10T19:21:32.431 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=deleted 2023-12-10T19:21:32.431 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:21:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_28 2023-12-10T19:21:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_28 2023-12-10T19:21:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_28 2023-12-10T19:21:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:32.470 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_28: (2) No such file or directory 2023-12-10T19:21:32.474 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image_id= 2023-12-10T19:21:32.474 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.1 2023-12-10T19:21:32.575 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_28 deleted 2023-12-10T19:21:32.575 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:32.576 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:32.576 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_28 2023-12-10T19:21:32.576 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=deleted 2023-12-10T19:21:32.576 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:32.576 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:21:32.576 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:21:32.576 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_28 2023-12-10T19:21:32.576 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:32.576 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:32.576 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_28 2023-12-10T19:21:32.577 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_28 2023-12-10T19:21:32.577 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:32.624 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_28: (2) No such file or directory 2023-12-10T19:21:32.628 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:21:32.628 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:32.628 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test deleted = deleted 2023-12-10T19:21:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:21:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_29 2023-12-10T19:21:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_present cluster1 mirror image_29 deleted 2023-12-10T19:21:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_29 2023-12-10T19:21:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=deleted 2023-12-10T19:21:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:21:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_29 2023-12-10T19:21:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_29 2023-12-10T19:21:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_29 2023-12-10T19:21:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:32.668 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_29: (2) No such file or directory 2023-12-10T19:21:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image_id= 2023-12-10T19:21:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.1 2023-12-10T19:21:32.773 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_29 deleted 2023-12-10T19:21:32.774 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:32.774 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:32.774 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_29 2023-12-10T19:21:32.774 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=deleted 2023-12-10T19:21:32.774 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:32.774 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:21:32.774 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:21:32.774 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_29 2023-12-10T19:21:32.774 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:32.774 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:32.774 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_29 2023-12-10T19:21:32.774 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_29 2023-12-10T19:21:32.774 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:32.820 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_29: (2) No such file or directory 2023-12-10T19:21:32.825 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:21:32.826 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:32.826 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test deleted = deleted 2023-12-10T19:21:32.826 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:21:32.826 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_30 2023-12-10T19:21:32.826 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_present cluster1 mirror image_30 deleted 2023-12-10T19:21:32.826 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:32.826 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:32.826 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_30 2023-12-10T19:21:32.826 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=deleted 2023-12-10T19:21:32.826 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:32.826 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:21:32.826 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:32.826 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_30 2023-12-10T19:21:32.827 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:32.827 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:32.827 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_30 2023-12-10T19:21:32.827 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_30 2023-12-10T19:21:32.827 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:32.864 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_30: (2) No such file or directory 2023-12-10T19:21:32.868 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image_id= 2023-12-10T19:21:32.868 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.1 2023-12-10T19:21:32.970 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_30 deleted 2023-12-10T19:21:32.970 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:32.970 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:32.970 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_30 2023-12-10T19:21:32.970 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=deleted 2023-12-10T19:21:32.970 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:32.970 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:21:32.970 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:21:32.970 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_30 2023-12-10T19:21:32.971 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:32.971 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:32.971 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_30 2023-12-10T19:21:32.971 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_30 2023-12-10T19:21:32.971 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:33.016 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_30: (2) No such file or directory 2023-12-10T19:21:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:21:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test deleted = deleted 2023-12-10T19:21:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:21:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_31 2023-12-10T19:21:33.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_present cluster1 mirror image_31 deleted 2023-12-10T19:21:33.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:33.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:33.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_31 2023-12-10T19:21:33.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=deleted 2023-12-10T19:21:33.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:33.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:21:33.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:33.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_31 2023-12-10T19:21:33.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:33.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:33.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_31 2023-12-10T19:21:33.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_31 2023-12-10T19:21:33.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:33.061 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_31: (2) No such file or directory 2023-12-10T19:21:33.064 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image_id= 2023-12-10T19:21:33.064 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.1 2023-12-10T19:21:33.166 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_31 deleted 2023-12-10T19:21:33.167 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:33.167 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:33.167 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_31 2023-12-10T19:21:33.167 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=deleted 2023-12-10T19:21:33.167 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:33.167 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:21:33.167 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:21:33.167 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_31 2023-12-10T19:21:33.167 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:33.167 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:33.167 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_31 2023-12-10T19:21:33.167 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_31 2023-12-10T19:21:33.167 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:33.212 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_31: (2) No such file or directory 2023-12-10T19:21:33.216 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:21:33.216 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:33.216 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test deleted = deleted 2023-12-10T19:21:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:21:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_32 2023-12-10T19:21:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_present cluster1 mirror image_32 deleted 2023-12-10T19:21:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_32 2023-12-10T19:21:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=deleted 2023-12-10T19:21:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:21:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_32 2023-12-10T19:21:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_32 2023-12-10T19:21:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_32 2023-12-10T19:21:33.218 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:33.255 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_32: (2) No such file or directory 2023-12-10T19:21:33.259 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image_id= 2023-12-10T19:21:33.259 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.1 2023-12-10T19:21:33.361 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_32 deleted 2023-12-10T19:21:33.361 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:33.361 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:33.361 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_32 2023-12-10T19:21:33.361 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=deleted 2023-12-10T19:21:33.361 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:33.361 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:21:33.361 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:21:33.361 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_32 2023-12-10T19:21:33.362 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:33.362 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:33.362 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_32 2023-12-10T19:21:33.362 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_32 2023-12-10T19:21:33.362 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:33.409 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_32: (2) No such file or directory 2023-12-10T19:21:33.413 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:21:33.413 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:33.413 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test deleted = deleted 2023-12-10T19:21:33.413 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:21:33.413 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_33 2023-12-10T19:21:33.414 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_present cluster1 mirror image_33 deleted 2023-12-10T19:21:33.414 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:33.414 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:33.414 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_33 2023-12-10T19:21:33.414 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=deleted 2023-12-10T19:21:33.414 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:33.414 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:21:33.414 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:33.414 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_33 2023-12-10T19:21:33.414 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:33.414 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:33.414 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_33 2023-12-10T19:21:33.414 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_33 2023-12-10T19:21:33.414 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:33.451 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_33: (2) No such file or directory 2023-12-10T19:21:33.455 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image_id= 2023-12-10T19:21:33.455 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.1 2023-12-10T19:21:33.556 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_33 deleted 2023-12-10T19:21:33.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:33.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:33.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_33 2023-12-10T19:21:33.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=deleted 2023-12-10T19:21:33.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:33.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:21:33.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:21:33.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_33 2023-12-10T19:21:33.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:33.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:33.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_33 2023-12-10T19:21:33.558 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_33 2023-12-10T19:21:33.558 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_33: (2) No such file or directory 2023-12-10T19:21:33.606 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:21:33.606 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:33.606 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test deleted = deleted 2023-12-10T19:21:33.606 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:21:33.607 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_34 2023-12-10T19:21:33.607 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_present cluster1 mirror image_34 deleted 2023-12-10T19:21:33.607 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:33.607 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:33.607 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_34 2023-12-10T19:21:33.607 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=deleted 2023-12-10T19:21:33.607 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:33.607 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:21:33.607 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:33.607 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_34 2023-12-10T19:21:33.607 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:33.607 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:33.607 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_34 2023-12-10T19:21:33.607 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_34 2023-12-10T19:21:33.608 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:33.647 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_34: (2) No such file or directory 2023-12-10T19:21:33.650 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image_id= 2023-12-10T19:21:33.650 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.1 2023-12-10T19:21:33.752 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_34 deleted 2023-12-10T19:21:33.752 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:33.753 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:33.753 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_34 2023-12-10T19:21:33.753 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=deleted 2023-12-10T19:21:33.753 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:33.753 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:21:33.753 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:21:33.753 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_34 2023-12-10T19:21:33.753 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:33.753 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:33.753 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_34 2023-12-10T19:21:33.753 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_34 2023-12-10T19:21:33.753 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:33.797 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_34: (2) No such file or directory 2023-12-10T19:21:33.801 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:21:33.801 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:33.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test deleted = deleted 2023-12-10T19:21:33.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:21:33.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_35 2023-12-10T19:21:33.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_present cluster1 mirror image_35 deleted 2023-12-10T19:21:33.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:33.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:33.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_35 2023-12-10T19:21:33.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=deleted 2023-12-10T19:21:33.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:33.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:21:33.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:33.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_35 2023-12-10T19:21:33.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:33.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:33.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_35 2023-12-10T19:21:33.803 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_35 2023-12-10T19:21:33.803 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_35: (2) No such file or directory 2023-12-10T19:21:33.844 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image_id= 2023-12-10T19:21:33.844 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.1 2023-12-10T19:21:33.946 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_35 deleted 2023-12-10T19:21:33.946 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:33.946 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:33.946 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_35 2023-12-10T19:21:33.946 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=deleted 2023-12-10T19:21:33.946 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:33.946 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:21:33.946 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:21:33.946 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_35 2023-12-10T19:21:33.946 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:33.946 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:33.947 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_35 2023-12-10T19:21:33.947 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_35 2023-12-10T19:21:33.947 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:33.991 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_35: (2) No such file or directory 2023-12-10T19:21:33.996 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:21:33.996 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:33.996 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test deleted = deleted 2023-12-10T19:21:33.996 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:21:33.996 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_36 2023-12-10T19:21:33.996 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_present cluster1 mirror image_36 deleted 2023-12-10T19:21:33.996 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:33.996 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:33.996 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_36 2023-12-10T19:21:33.996 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=deleted 2023-12-10T19:21:33.996 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:33.996 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:21:33.996 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:33.997 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_36 2023-12-10T19:21:33.997 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:33.997 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:33.997 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_36 2023-12-10T19:21:33.997 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_36 2023-12-10T19:21:33.997 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:34.036 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_36: (2) No such file or directory 2023-12-10T19:21:34.040 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image_id= 2023-12-10T19:21:34.040 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.1 2023-12-10T19:21:34.142 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_36 deleted 2023-12-10T19:21:34.142 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:34.142 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:34.142 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_36 2023-12-10T19:21:34.142 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=deleted 2023-12-10T19:21:34.142 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:34.142 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:21:34.142 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:21:34.143 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_36 2023-12-10T19:21:34.143 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:34.143 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:34.143 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_36 2023-12-10T19:21:34.143 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_36 2023-12-10T19:21:34.143 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:34.189 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_36: (2) No such file or directory 2023-12-10T19:21:34.194 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:21:34.194 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:34.194 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test deleted = deleted 2023-12-10T19:21:34.194 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:21:34.194 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_37 2023-12-10T19:21:34.194 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_present cluster1 mirror image_37 deleted 2023-12-10T19:21:34.194 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:34.194 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:34.195 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_37 2023-12-10T19:21:34.195 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=deleted 2023-12-10T19:21:34.195 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:34.195 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:21:34.195 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:34.195 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_37 2023-12-10T19:21:34.195 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:34.195 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:34.195 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_37 2023-12-10T19:21:34.195 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_37 2023-12-10T19:21:34.195 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:34.233 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_37: (2) No such file or directory 2023-12-10T19:21:34.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image_id= 2023-12-10T19:21:34.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.1 2023-12-10T19:21:34.338 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_37 deleted 2023-12-10T19:21:34.339 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:34.339 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:34.339 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_37 2023-12-10T19:21:34.339 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=deleted 2023-12-10T19:21:34.339 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:34.339 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:21:34.339 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:21:34.339 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_37 2023-12-10T19:21:34.339 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:34.339 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:34.339 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_37 2023-12-10T19:21:34.339 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_37 2023-12-10T19:21:34.339 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:34.386 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_37: (2) No such file or directory 2023-12-10T19:21:34.390 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:21:34.391 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:34.391 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test deleted = deleted 2023-12-10T19:21:34.391 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:21:34.391 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_38 2023-12-10T19:21:34.391 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_present cluster1 mirror image_38 deleted 2023-12-10T19:21:34.391 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:34.391 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:34.391 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_38 2023-12-10T19:21:34.391 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=deleted 2023-12-10T19:21:34.391 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:34.391 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:21:34.391 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:34.391 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_38 2023-12-10T19:21:34.391 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:34.392 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:34.392 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_38 2023-12-10T19:21:34.392 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_38 2023-12-10T19:21:34.392 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:34.428 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_38: (2) No such file or directory 2023-12-10T19:21:34.431 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image_id= 2023-12-10T19:21:34.431 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.1 2023-12-10T19:21:34.533 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_38 deleted 2023-12-10T19:21:34.533 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:34.533 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:34.533 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_38 2023-12-10T19:21:34.533 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=deleted 2023-12-10T19:21:34.533 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:34.533 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:21:34.534 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:21:34.534 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_38 2023-12-10T19:21:34.534 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:34.534 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:34.534 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_38 2023-12-10T19:21:34.534 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_38 2023-12-10T19:21:34.534 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:34.581 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_38: (2) No such file or directory 2023-12-10T19:21:34.586 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:21:34.586 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:34.586 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test deleted = deleted 2023-12-10T19:21:34.586 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:21:34.586 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_39 2023-12-10T19:21:34.586 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_present cluster1 mirror image_39 deleted 2023-12-10T19:21:34.586 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:34.586 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:34.586 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_39 2023-12-10T19:21:34.586 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=deleted 2023-12-10T19:21:34.586 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:34.586 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:21:34.586 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:34.587 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_39 2023-12-10T19:21:34.587 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:34.587 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:34.587 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_39 2023-12-10T19:21:34.587 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_39 2023-12-10T19:21:34.587 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:34.625 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_39: (2) No such file or directory 2023-12-10T19:21:34.628 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image_id= 2023-12-10T19:21:34.629 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.1 2023-12-10T19:21:34.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_39 deleted 2023-12-10T19:21:34.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:34.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:34.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_39 2023-12-10T19:21:34.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=deleted 2023-12-10T19:21:34.731 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:34.731 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:21:34.731 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:21:34.731 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_39 2023-12-10T19:21:34.731 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:34.731 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:34.731 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_39 2023-12-10T19:21:34.731 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_39 2023-12-10T19:21:34.731 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:34.775 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_39: (2) No such file or directory 2023-12-10T19:21:34.780 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:21:34.780 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:34.780 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test deleted = deleted 2023-12-10T19:21:34.780 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:21:34.780 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_40 2023-12-10T19:21:34.780 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_present cluster1 mirror image_40 deleted 2023-12-10T19:21:34.780 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:34.780 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:34.780 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_40 2023-12-10T19:21:34.780 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=deleted 2023-12-10T19:21:34.780 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:34.781 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:21:34.781 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:34.781 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_40 2023-12-10T19:21:34.781 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:34.781 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:34.781 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_40 2023-12-10T19:21:34.781 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_40 2023-12-10T19:21:34.781 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:34.818 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_40: (2) No such file or directory 2023-12-10T19:21:34.822 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image_id= 2023-12-10T19:21:34.822 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.1 2023-12-10T19:21:34.924 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_40 deleted 2023-12-10T19:21:34.924 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:34.924 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:34.925 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_40 2023-12-10T19:21:34.925 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=deleted 2023-12-10T19:21:34.925 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:34.925 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:21:34.925 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:21:34.925 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_40 2023-12-10T19:21:34.925 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:34.925 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:34.925 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_40 2023-12-10T19:21:34.925 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_40 2023-12-10T19:21:34.925 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:34.970 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_40: (2) No such file or directory 2023-12-10T19:21:34.975 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:21:34.976 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:34.976 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test deleted = deleted 2023-12-10T19:21:34.976 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:21:34.976 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_41 2023-12-10T19:21:34.976 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_present cluster1 mirror image_41 deleted 2023-12-10T19:21:34.976 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:34.976 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:34.976 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_41 2023-12-10T19:21:34.976 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=deleted 2023-12-10T19:21:34.976 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:34.976 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:21:34.976 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:34.976 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_41 2023-12-10T19:21:34.976 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:34.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:34.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_41 2023-12-10T19:21:34.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_41 2023-12-10T19:21:34.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:35.014 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_41: (2) No such file or directory 2023-12-10T19:21:35.018 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image_id= 2023-12-10T19:21:35.018 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.1 2023-12-10T19:21:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_41 deleted 2023-12-10T19:21:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_41 2023-12-10T19:21:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=deleted 2023-12-10T19:21:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:21:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:21:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_41 2023-12-10T19:21:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:35.121 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_41 2023-12-10T19:21:35.121 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_41 2023-12-10T19:21:35.121 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:35.168 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_41: (2) No such file or directory 2023-12-10T19:21:35.173 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:21:35.173 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:35.173 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test deleted = deleted 2023-12-10T19:21:35.173 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:21:35.174 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_42 2023-12-10T19:21:35.174 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_present cluster1 mirror image_42 deleted 2023-12-10T19:21:35.174 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:35.174 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:35.174 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_42 2023-12-10T19:21:35.174 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=deleted 2023-12-10T19:21:35.174 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:35.174 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:21:35.174 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:35.174 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_42 2023-12-10T19:21:35.174 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:35.174 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:35.174 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_42 2023-12-10T19:21:35.175 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_42 2023-12-10T19:21:35.175 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:35.211 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_42: (2) No such file or directory 2023-12-10T19:21:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image_id= 2023-12-10T19:21:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.1 2023-12-10T19:21:35.316 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_42 deleted 2023-12-10T19:21:35.317 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:35.317 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:35.317 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_42 2023-12-10T19:21:35.317 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=deleted 2023-12-10T19:21:35.317 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:35.317 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:21:35.317 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:21:35.317 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_42 2023-12-10T19:21:35.317 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:35.317 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:35.317 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_42 2023-12-10T19:21:35.317 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_42 2023-12-10T19:21:35.317 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:35.362 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_42: (2) No such file or directory 2023-12-10T19:21:35.367 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:21:35.367 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:35.367 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test deleted = deleted 2023-12-10T19:21:35.367 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:21:35.367 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_43 2023-12-10T19:21:35.367 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_present cluster1 mirror image_43 deleted 2023-12-10T19:21:35.367 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:35.367 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:35.367 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_43 2023-12-10T19:21:35.367 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=deleted 2023-12-10T19:21:35.367 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:35.367 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:21:35.367 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:35.368 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_43 2023-12-10T19:21:35.368 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:35.368 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:35.368 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_43 2023-12-10T19:21:35.368 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_43 2023-12-10T19:21:35.368 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:35.406 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_43: (2) No such file or directory 2023-12-10T19:21:35.409 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image_id= 2023-12-10T19:21:35.409 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.1 2023-12-10T19:21:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_43 deleted 2023-12-10T19:21:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_43 2023-12-10T19:21:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=deleted 2023-12-10T19:21:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:21:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:21:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_43 2023-12-10T19:21:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:35.512 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_43 2023-12-10T19:21:35.512 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_43 2023-12-10T19:21:35.512 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:35.555 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_43: (2) No such file or directory 2023-12-10T19:21:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:21:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test deleted = deleted 2023-12-10T19:21:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:21:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_44 2023-12-10T19:21:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_present cluster1 mirror image_44 deleted 2023-12-10T19:21:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_44 2023-12-10T19:21:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=deleted 2023-12-10T19:21:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:35.561 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:21:35.561 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:35.561 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_44 2023-12-10T19:21:35.561 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:35.561 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:35.561 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_44 2023-12-10T19:21:35.561 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_44 2023-12-10T19:21:35.561 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_44: (2) No such file or directory 2023-12-10T19:21:35.602 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image_id= 2023-12-10T19:21:35.602 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.1 2023-12-10T19:21:35.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_44 deleted 2023-12-10T19:21:35.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:35.703 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:35.704 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_44 2023-12-10T19:21:35.704 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=deleted 2023-12-10T19:21:35.704 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:35.704 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:21:35.704 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:21:35.704 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_44 2023-12-10T19:21:35.704 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:35.704 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:35.704 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_44 2023-12-10T19:21:35.704 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_44 2023-12-10T19:21:35.704 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:35.751 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_44: (2) No such file or directory 2023-12-10T19:21:35.756 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:21:35.756 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:35.756 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test deleted = deleted 2023-12-10T19:21:35.756 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:21:35.756 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_45 2023-12-10T19:21:35.756 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_present cluster1 mirror image_45 deleted 2023-12-10T19:21:35.756 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:35.757 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:35.757 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_45 2023-12-10T19:21:35.757 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=deleted 2023-12-10T19:21:35.757 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:35.757 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:21:35.757 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:35.757 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_45 2023-12-10T19:21:35.757 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:35.757 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:35.757 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_45 2023-12-10T19:21:35.757 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_45 2023-12-10T19:21:35.757 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:35.796 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_45: (2) No such file or directory 2023-12-10T19:21:35.799 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image_id= 2023-12-10T19:21:35.799 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.1 2023-12-10T19:21:35.901 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_45 deleted 2023-12-10T19:21:35.901 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:35.901 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:35.901 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_45 2023-12-10T19:21:35.901 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=deleted 2023-12-10T19:21:35.901 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:35.902 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:21:35.902 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:21:35.902 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_45 2023-12-10T19:21:35.902 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:35.902 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:35.902 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_45 2023-12-10T19:21:35.902 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_45 2023-12-10T19:21:35.902 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:35.949 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_45: (2) No such file or directory 2023-12-10T19:21:35.953 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:21:35.954 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:35.954 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test deleted = deleted 2023-12-10T19:21:35.954 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:21:35.954 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_46 2023-12-10T19:21:35.954 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_present cluster1 mirror image_46 deleted 2023-12-10T19:21:35.954 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:35.954 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:35.954 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_46 2023-12-10T19:21:35.954 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=deleted 2023-12-10T19:21:35.954 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:35.954 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:21:35.954 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:35.954 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_46 2023-12-10T19:21:35.955 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:35.955 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:35.955 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_46 2023-12-10T19:21:35.955 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_46 2023-12-10T19:21:35.955 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:35.993 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_46: (2) No such file or directory 2023-12-10T19:21:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image_id= 2023-12-10T19:21:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.1 2023-12-10T19:21:36.099 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_46 deleted 2023-12-10T19:21:36.099 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:36.099 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:36.099 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_46 2023-12-10T19:21:36.099 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=deleted 2023-12-10T19:21:36.099 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:36.099 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:21:36.099 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:21:36.099 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_46 2023-12-10T19:21:36.099 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:36.099 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:36.099 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_46 2023-12-10T19:21:36.099 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_46 2023-12-10T19:21:36.100 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:36.143 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_46: (2) No such file or directory 2023-12-10T19:21:36.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:21:36.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:36.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test deleted = deleted 2023-12-10T19:21:36.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:21:36.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_47 2023-12-10T19:21:36.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_present cluster1 mirror image_47 deleted 2023-12-10T19:21:36.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:36.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:36.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_47 2023-12-10T19:21:36.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=deleted 2023-12-10T19:21:36.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:36.149 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:21:36.149 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:36.149 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_47 2023-12-10T19:21:36.149 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:36.149 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:36.149 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_47 2023-12-10T19:21:36.149 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_47 2023-12-10T19:21:36.149 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:36.186 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_47: (2) No such file or directory 2023-12-10T19:21:36.190 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image_id= 2023-12-10T19:21:36.190 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.1 2023-12-10T19:21:36.292 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_47 deleted 2023-12-10T19:21:36.292 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:36.292 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:36.292 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_47 2023-12-10T19:21:36.292 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=deleted 2023-12-10T19:21:36.292 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:36.292 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:21:36.292 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:21:36.292 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_47 2023-12-10T19:21:36.292 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:36.292 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:36.292 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_47 2023-12-10T19:21:36.293 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_47 2023-12-10T19:21:36.293 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:36.337 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_47: (2) No such file or directory 2023-12-10T19:21:36.342 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:21:36.343 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:36.343 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test deleted = deleted 2023-12-10T19:21:36.343 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:21:36.343 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_48 2023-12-10T19:21:36.343 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_present cluster1 mirror image_48 deleted 2023-12-10T19:21:36.343 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:36.343 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:36.343 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_48 2023-12-10T19:21:36.343 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=deleted 2023-12-10T19:21:36.343 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:36.343 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:21:36.343 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:36.343 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_48 2023-12-10T19:21:36.343 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:36.344 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:36.344 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_48 2023-12-10T19:21:36.344 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_48 2023-12-10T19:21:36.344 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:36.381 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_48: (2) No such file or directory 2023-12-10T19:21:36.384 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image_id= 2023-12-10T19:21:36.385 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.1 2023-12-10T19:21:36.486 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_48 deleted 2023-12-10T19:21:36.486 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:36.486 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:36.486 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_48 2023-12-10T19:21:36.486 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=deleted 2023-12-10T19:21:36.487 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:36.487 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:21:36.487 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:21:36.487 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_48 2023-12-10T19:21:36.487 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:36.487 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:36.487 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_48 2023-12-10T19:21:36.488 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_48 2023-12-10T19:21:36.488 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:36.533 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_48: (2) No such file or directory 2023-12-10T19:21:36.539 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:21:36.539 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:36.539 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test deleted = deleted 2023-12-10T19:21:36.540 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:21:36.540 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_49 2023-12-10T19:21:36.540 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_present cluster1 mirror image_49 deleted 2023-12-10T19:21:36.540 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:36.540 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:36.540 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_49 2023-12-10T19:21:36.540 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=deleted 2023-12-10T19:21:36.540 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:36.541 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:21:36.541 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:36.541 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_49 2023-12-10T19:21:36.541 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:36.541 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:36.541 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_49 2023-12-10T19:21:36.541 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_49 2023-12-10T19:21:36.541 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:36.576 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_49: (2) No such file or directory 2023-12-10T19:21:36.579 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image_id= 2023-12-10T19:21:36.579 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.1 2023-12-10T19:21:36.681 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_49 deleted 2023-12-10T19:21:36.681 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:36.682 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:36.682 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_49 2023-12-10T19:21:36.682 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=deleted 2023-12-10T19:21:36.682 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:36.682 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:21:36.682 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:21:36.682 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_49 2023-12-10T19:21:36.682 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:36.682 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:36.682 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_49 2023-12-10T19:21:36.682 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_49 2023-12-10T19:21:36.682 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:36.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_49: (2) No such file or directory 2023-12-10T19:21:36.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:21:36.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:36.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test deleted = deleted 2023-12-10T19:21:36.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:21:36.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_50 2023-12-10T19:21:36.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_present cluster1 mirror image_50 deleted 2023-12-10T19:21:36.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:36.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:36.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_50 2023-12-10T19:21:36.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=deleted 2023-12-10T19:21:36.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:36.733 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:21:36.733 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:36.733 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_50 2023-12-10T19:21:36.733 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:36.733 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:36.733 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_50 2023-12-10T19:21:36.733 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_50 2023-12-10T19:21:36.733 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:36.770 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_50: (2) No such file or directory 2023-12-10T19:21:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image_id= 2023-12-10T19:21:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.1 2023-12-10T19:21:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_50 deleted 2023-12-10T19:21:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_50 2023-12-10T19:21:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=deleted 2023-12-10T19:21:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:21:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:21:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_50 2023-12-10T19:21:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_50 2023-12-10T19:21:36.876 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_50 2023-12-10T19:21:36.876 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:36.920 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_50: (2) No such file or directory 2023-12-10T19:21:36.924 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:21:36.924 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:36.924 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test deleted = deleted 2023-12-10T19:21:36.924 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:21:36.924 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ testlog TEST: delete images during bootstrap 2023-12-10T19:21:36.924 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ tee -a /tmp/tmp.qNjRyMq75D/rbd-mirror.test.log 2023-12-10T19:21:36.924 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ date +%F %T 2023-12-10T19:21:36.925 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo 2023-12-10 19:21:36 TEST: delete images during bootstrap 2023-12-10T19:21:36.925 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:2023-12-10 19:21:36 TEST: delete images during bootstrap 2023-12-10T19:21:36.925 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_pool_mirror_mode cluster1 mirror image 2023-12-10T19:21:36.925 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:36.925 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:36.925 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=image 2023-12-10T19:21:36.926 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 mirror pool enable mirror image 2023-12-10T19:21:36.964 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd: mirroring is already configured for image mode 2023-12-10T19:21:36.967 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_pool_mirror_mode cluster2 mirror image 2023-12-10T19:21:36.967 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:21:36.967 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:36.967 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=image 2023-12-10T19:21:36.967 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 mirror pool enable mirror image 2023-12-10T19:21:37.002 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd: mirroring is already configured for image mode 2023-12-10T19:21:37.005 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ start_mirror cluster1 2023-12-10T19:21:37.006 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:37.006 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T19:21:37.006 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1 cluster instance 2023-12-10T19:21:37.006 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1 2023-12-10T19:21:37.006 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T19:21:37.006 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T19:21:37.006 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:21:37.006 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=cluster1 2023-12-10T19:21:37.006 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ cluster1 = cluster1 ] 2023-12-10T19:21:37.006 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:21:37.006 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T19:21:37.006 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T19:21:37.006 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T19:21:37.006 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T19:21:37.007 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 1 2023-12-10T19:21:37.007 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 2023-12-10T19:21:37.007 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=test 2023-12-10T19:21:37.007 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 1 10 2023-12-10T19:21:37.007 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_1 2023-12-10T19:21:37.007 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image cluster2 mirror image_1 512M 2023-12-10T19:21:37.007 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:21:37.008 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:21:37.008 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:37.008 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:21:37.008 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_1 2023-12-10T19:21:37.008 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:21:37.008 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local size=128 2023-12-10T19:21:37.008 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n 512M ] 2023-12-10T19:21:37.008 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ size=512M 2023-12-10T19:21:37.008 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:21:37.008 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/image_1 2023-12-10T19:21:37.059 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ enable_mirror cluster2 mirror image_1 2023-12-10T19:21:37.059 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:21:37.059 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:37.059 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_1 2023-12-10T19:21:37.059 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:21:37.059 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 snapshot 2023-12-10T19:21:37.873 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirroring enabled 2023-12-10T19:21:37.883 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2023-12-10T19:21:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_1': 2023-12-10T19:21:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 512 MiB in 128 objects 2023-12-10T19:21:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:21:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 1 2023-12-10T19:21:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 18e415b965de 2023-12-10T19:21:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.18e415b965de 2023-12-10T19:21:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:21:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering 2023-12-10T19:21:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:21:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:21:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:21:37 2023 2023-12-10T19:21:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:21:37 2023 2023-12-10T19:21:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:21:37 2023 2023-12-10T19:21:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:21:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:21:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: 8b3f7800-15da-4697-8659-d2aa7e432571 2023-12-10T19:21:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: true 2023-12-10T19:21:38.144 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stress_write_image cluster2 mirror image_1 2023-12-10T19:21:38.144 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:21:38.144 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:38.144 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_1 2023-12-10T19:21:38.145 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-10T19:21:38.146 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local duration=6 2023-12-10T19:21:38.146 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T19:21:38.146 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2023-12-10T19:21:38.413 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:100 IOs, 42455040 bytes 2023-12-10T19:21:38.602 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:200 IOs, 79772672 bytes 2023-12-10T19:21:38.921 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:300 IOs, 173024256 bytes 2023-12-10T19:21:39.215 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:400 IOs, 264816640 bytes 2023-12-10T19:21:39.388 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:500 IOs, 300269568 bytes 2023-12-10T19:21:39.520 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:600 IOs, 312020992 bytes 2023-12-10T19:21:39.666 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:700 IOs, 328786944 bytes 2023-12-10T19:21:39.825 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:800 IOs, 358300672 bytes 2023-12-10T19:21:39.886 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-12-10T19:21:39.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-12-10T19:21:39.887 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2023-12-10T19:21:39.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr:2023-12-10T19:21:39.888+0000 7f0ad1480640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 68487) UID: 1000 2023-12-10T19:21:40.073 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:900 IOs, 418932736 bytes 2023-12-10T19:21:40.224 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1000 IOs, 443681792 bytes 2023-12-10T19:21:40.383 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1100 IOs, 472162304 bytes 2023-12-10T19:21:40.672 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1200 IOs, 561363968 bytes 2023-12-10T19:21:40.841 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1300 IOs, 593421312 bytes 2023-12-10T19:21:41.064 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1400 IOs, 648260608 bytes 2023-12-10T19:21:41.416 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1500 IOs, 762038272 bytes 2023-12-10T19:21:41.600 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1600 IOs, 804268032 bytes 2023-12-10T19:21:41.764 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1700 IOs, 844100608 bytes 2023-12-10T19:21:41.981 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1800 IOs, 881972224 bytes 2023-12-10T19:21:42.193 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1900 IOs, 929924096 bytes 2023-12-10T19:21:42.340 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2000 IOs, 952896512 bytes 2023-12-10T19:21:42.499 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2100 IOs, 981809152 bytes 2023-12-10T19:21:42.711 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2200 IOs, 1031789568 bytes 2023-12-10T19:21:42.991 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2300 IOs, 1118703616 bytes 2023-12-10T19:21:43.224 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2400 IOs, 1183191040 bytes 2023-12-10T19:21:43.649 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2500 IOs, 1273330688 bytes 2023-12-10T19:21:43.803 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2600 IOs, 1286528000 bytes 2023-12-10T19:21:44.026 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2700 IOs, 1360901120 bytes 2023-12-10T19:21:44.153 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=124 2023-12-10T19:21:44.153 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T19:21:44.153 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 124 -eq 124 ] 2023-12-10T19:21:44.153 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:21:44.153 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_present cluster1 mirror image_1 present 2023-12-10T19:21:44.153 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:44.153 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:44.153 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_1 2023-12-10T19:21:44.153 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=present 2023-12-10T19:21:44.154 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:44.154 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:21:44.154 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:44.154 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_1 2023-12-10T19:21:44.154 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:44.154 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:44.154 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_1 2023-12-10T19:21:44.154 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_1 2023-12-10T19:21:44.154 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image_id=128b8127c355 2023-12-10T19:21:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.1 2023-12-10T19:21:44.309 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_1 present 128b8127c355 2023-12-10T19:21:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_1 2023-12-10T19:21:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=present 2023-12-10T19:21:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id=128b8127c355 2023-12-10T19:21:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:21:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:21:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_1 2023-12-10T19:21:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_1 2023-12-10T19:21:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_1 2023-12-10T19:21:44.311 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:44.371 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id=128b8127c355 2023-12-10T19:21:44.371 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 128b8127c355 2023-12-10T19:21:44.371 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -z 128b8127c355 -o 128b8127c355 = 128b8127c355 2023-12-10T19:21:44.371 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=present 2023-12-10T19:21:44.371 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test present = present 2023-12-10T19:21:44.371 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:21:44.371 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ disable_mirror cluster2 mirror image_1 2023-12-10T19:21:44.372 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:21:44.372 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:44.372 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_1 2023-12-10T19:21:44.372 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_1 2023-12-10T19:21:44.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi076.stderr:2023-12-10T19:21:44.872+0000 7f3eaba39640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55b33bf13400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=368 2023-12-10T19:21:44.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi076.stderr:2023-12-10T19:21:44.872+0000 7f3eaba39640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55b33bf13400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=372, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=f4a845a0-65d8-4baa-abd4-fd2893f9af83, primary_snap_id=170, last_copied_object_number=128, snap_seqs={368=18446744073709551614}] 2023-12-10T19:21:45.888 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-12-10T19:21:45.888 INFO:teuthology.orchestra.run:waiting for 600 2023-12-10T19:21:45.889 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-12-10T19:21:45.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-10T19:21:45.889 DEBUG:teuthology.orchestra.run.smithi076:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-10T19:21:45.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-10T19:21:45.892 INFO:tasks.rbd_mirror_thrash:waiting for 87 secs before thrashing 2023-12-10T19:21:49.878 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirroring disabled 2023-12-10T19:21:49.893 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2023-12-10T19:21:49.893 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:49.893 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:49.893 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_1 2023-12-10T19:21:49.894 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=deleted 2023-12-10T19:21:49.894 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:49.894 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:21:49.894 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:49.895 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_1 2023-12-10T19:21:49.895 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:49.895 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:49.895 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_1 2023-12-10T19:21:49.895 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_1 2023-12-10T19:21:49.895 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:49.941 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_1: (2) No such file or directory 2023-12-10T19:21:49.945 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image_id= 2023-12-10T19:21:49.945 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.1 2023-12-10T19:21:50.047 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_1 deleted 2023-12-10T19:21:50.047 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:50.047 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:50.047 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_1 2023-12-10T19:21:50.047 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=deleted 2023-12-10T19:21:50.048 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:21:50.048 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:21:50.048 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:21:50.048 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_1 2023-12-10T19:21:50.048 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:21:50.048 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:50.048 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_1 2023-12-10T19:21:50.048 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_1 2023-12-10T19:21:50.048 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:21:50.089 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_1: (2) No such file or directory 2023-12-10T19:21:50.092 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:21:50.092 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:21:50.092 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test deleted = deleted 2023-12-10T19:21:50.093 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:21:50.093 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ purge_snapshots cluster2 mirror image_1 2023-12-10T19:21:50.093 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:21:50.093 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:50.093 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_1 2023-12-10T19:21:50.093 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap purge mirror/image_1 2023-12-10T19:21:50.141 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image_retry cluster2 mirror image_1 2023-12-10T19:21:50.141 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:21:50.141 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:50.141 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_1 2023-12-10T19:21:50.141 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:21:50.142 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_1 2023-12-10T19:21:50.142 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:21:50.142 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:50.142 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_1 2023-12-10T19:21:50.142 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2023-12-10T19:21:50.190 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2023-12-10T19:21:50.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 0% complete...failed. 2023-12-10T19:21:50.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error: image still has watchers 2023-12-10T19:21:50.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-10T19:21:50.240 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:21:51.241 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_1 2023-12-10T19:21:51.241 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:21:51.242 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:51.242 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_1 2023-12-10T19:21:51.242 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2023-12-10T19:21:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2023-12-10T19:21:51.348 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 0% complete...failed. 2023-12-10T19:21:51.349 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error: image still has watchers 2023-12-10T19:21:51.349 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-10T19:21:51.352 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 2 2023-12-10T19:21:53.354 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_1 2023-12-10T19:21:53.354 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:21:53.354 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:53.354 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_1 2023-12-10T19:21:53.354 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2023-12-10T19:21:53.412 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2023-12-10T19:21:53.459 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 0% complete...failed. 2023-12-10T19:21:53.459 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error: image still has watchers 2023-12-10T19:21:53.459 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-10T19:21:53.463 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 4 2023-12-10T19:21:55.880 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.8256946500730958) > weight (0.25) 2023-12-10T19:21:55.880 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.966449936917164) > weight (0.25) 2023-12-10T19:21:55.880 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.4119284182752019) > weight (0.25) 2023-12-10T19:21:55.880 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2023-12-10T19:21:55.881 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2023-12-10T19:21:55.881 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2023-12-10T19:21:55.882 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi076.stderr:2023-12-10T19:21:55.887+0000 7f558fafc640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 43283) UID: 1000 2023-12-10T19:21:57.465 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_1 2023-12-10T19:21:57.465 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:21:57.465 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:21:57.465 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_1 2023-12-10T19:21:57.465 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2023-12-10T19:21:57.521 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2023-12-10T19:21:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 0% complete...failed. 2023-12-10T19:21:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error: image still has watchers 2023-12-10T19:21:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-10T19:21:57.571 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 8 2023-12-10T19:21:59.882 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2023-12-10T19:21:59.882 INFO:teuthology.orchestra.run:waiting for 600 2023-12-10T19:21:59.883 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2023-12-10T19:21:59.883 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2023-12-10T19:21:59.883 DEBUG:teuthology.orchestra.run.smithi076:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2023-12-10T19:21:59.886 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2023-12-10T19:21:59.887 INFO:tasks.rbd_mirror_thrash:waiting for 94 secs before thrashing 2023-12-10T19:22:05.573 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_1 2023-12-10T19:22:05.573 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:22:05.573 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:22:05.573 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_1 2023-12-10T19:22:05.574 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2023-12-10T19:22:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2023-12-10T19:22:05.678 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 0% complete...failed. 2023-12-10T19:22:05.678 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error: image still has watchers 2023-12-10T19:22:05.678 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-10T19:22:05.681 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 16 2023-12-10T19:22:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_1 2023-12-10T19:22:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:22:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:22:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_1 2023-12-10T19:22:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2023-12-10T19:22:21.742 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2023-12-10T19:22:22.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-10T19:22:22.026 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:22:22.026 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_2 2023-12-10T19:22:22.026 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image cluster2 mirror image_2 512M 2023-12-10T19:22:22.026 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:22:22.026 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:22:22.026 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:22:22.026 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:22:22.026 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_2 2023-12-10T19:22:22.026 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:22:22.026 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local size=128 2023-12-10T19:22:22.027 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n 512M ] 2023-12-10T19:22:22.027 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ size=512M 2023-12-10T19:22:22.027 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:22:22.027 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/image_2 2023-12-10T19:22:22.074 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ enable_mirror cluster2 mirror image_2 2023-12-10T19:22:22.075 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:22:22.075 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:22:22.075 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_2 2023-12-10T19:22:22.075 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:22:22.075 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 snapshot 2023-12-10T19:22:23.115 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirroring enabled 2023-12-10T19:22:23.129 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2023-12-10T19:22:23.174 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_2': 2023-12-10T19:22:23.174 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 512 MiB in 128 objects 2023-12-10T19:22:23.175 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:22:23.175 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 1 2023-12-10T19:22:23.175 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 192a717b6b11 2023-12-10T19:22:23.175 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.192a717b6b11 2023-12-10T19:22:23.175 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:22:23.175 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering 2023-12-10T19:22:23.175 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:22:23.175 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:22:23.175 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:22:22 2023 2023-12-10T19:22:23.175 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:22:22 2023 2023-12-10T19:22:23.175 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:22:22 2023 2023-12-10T19:22:23.175 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:22:23.175 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:22:23.175 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: 2a42db63-d09e-491d-a2c3-7aa044220aae 2023-12-10T19:22:23.175 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: true 2023-12-10T19:22:23.179 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stress_write_image cluster2 mirror image_2 2023-12-10T19:22:23.179 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:22:23.179 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:22:23.179 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_2 2023-12-10T19:22:23.179 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-10T19:22:23.180 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local duration=6 2023-12-10T19:22:23.181 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T19:22:23.181 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2023-12-10T19:22:23.474 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:100 IOs, 67699712 bytes 2023-12-10T19:22:23.747 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:200 IOs, 142404608 bytes 2023-12-10T19:22:23.932 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:300 IOs, 186039296 bytes 2023-12-10T19:22:24.189 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:400 IOs, 272169984 bytes 2023-12-10T19:22:24.296 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:500 IOs, 272791552 bytes 2023-12-10T19:22:24.450 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:600 IOs, 298065920 bytes 2023-12-10T19:22:24.594 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:700 IOs, 321061888 bytes 2023-12-10T19:22:24.771 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:800 IOs, 353128448 bytes 2023-12-10T19:22:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:900 IOs, 386996224 bytes 2023-12-10T19:22:25.113 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1000 IOs, 428979200 bytes 2023-12-10T19:22:25.345 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1100 IOs, 502591488 bytes 2023-12-10T19:22:25.522 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1200 IOs, 514726912 bytes 2023-12-10T19:22:25.647 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1300 IOs, 523995136 bytes 2023-12-10T19:22:25.786 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1400 IOs, 544161792 bytes 2023-12-10T19:22:25.965 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1500 IOs, 574052352 bytes 2023-12-10T19:22:26.378 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1600 IOs, 644446208 bytes 2023-12-10T19:22:26.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1700 IOs, 708415488 bytes 2023-12-10T19:22:26.858 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1800 IOs, 752847872 bytes 2023-12-10T19:22:27.069 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1900 IOs, 809729024 bytes 2023-12-10T19:22:27.283 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2000 IOs, 859373568 bytes 2023-12-10T19:22:27.421 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2100 IOs, 875420672 bytes 2023-12-10T19:22:27.684 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2200 IOs, 948633600 bytes 2023-12-10T19:22:27.814 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2300 IOs, 957264896 bytes 2023-12-10T19:22:28.042 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2400 IOs, 1024501760 bytes 2023-12-10T19:22:28.220 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2500 IOs, 1065723904 bytes 2023-12-10T19:22:28.463 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2600 IOs, 1135497216 bytes 2023-12-10T19:22:28.570 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2700 IOs, 1136112640 bytes 2023-12-10T19:22:28.762 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2800 IOs, 1179492352 bytes 2023-12-10T19:22:28.942 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2900 IOs, 1234486272 bytes 2023-12-10T19:22:29.110 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3000 IOs, 1259305984 bytes 2023-12-10T19:22:29.185 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=124 2023-12-10T19:22:29.185 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T19:22:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 124 -eq 124 ] 2023-12-10T19:22:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:22:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_present cluster1 mirror image_2 present 2023-12-10T19:22:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:22:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:22:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_2 2023-12-10T19:22:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=present 2023-12-10T19:22:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:22:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:22:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:22:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_2 2023-12-10T19:22:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:22:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:22:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_2 2023-12-10T19:22:29.187 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_2 2023-12-10T19:22:29.187 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:22:29.456 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image_id=128ba3aebae7 2023-12-10T19:22:29.457 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.1 2023-12-10T19:22:29.559 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_2 present 128ba3aebae7 2023-12-10T19:22:29.559 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:22:29.559 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:22:29.559 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_2 2023-12-10T19:22:29.559 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=present 2023-12-10T19:22:29.559 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id=128ba3aebae7 2023-12-10T19:22:29.559 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:22:29.559 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:22:29.559 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_2 2023-12-10T19:22:29.559 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:22:29.559 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:22:29.559 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_2 2023-12-10T19:22:29.559 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_2 2023-12-10T19:22:29.560 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:22:29.835 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id=128ba3aebae7 2023-12-10T19:22:29.835 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 128ba3aebae7 2023-12-10T19:22:29.835 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -z 128ba3aebae7 -o 128ba3aebae7 = 128ba3aebae7 2023-12-10T19:22:29.835 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=present 2023-12-10T19:22:29.835 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test present = present 2023-12-10T19:22:29.835 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:22:29.836 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ disable_mirror cluster2 mirror image_2 2023-12-10T19:22:29.836 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:22:29.836 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:22:29.836 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_2 2023-12-10T19:22:29.836 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_2 2023-12-10T19:22:30.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi076.stderr:2023-12-10T19:22:30.890+0000 7f3eaba39640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55b33bf13800 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=370 2023-12-10T19:22:30.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi076.stderr:2023-12-10T19:22:30.890+0000 7f3eaba39640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55b33bf13800 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=374, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=f4a845a0-65d8-4baa-abd4-fd2893f9af83, primary_snap_id=172, last_copied_object_number=128, snap_seqs={370=18446744073709551614}] 2023-12-10T19:22:35.899 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirroring disabled 2023-12-10T19:22:35.916 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2023-12-10T19:22:35.916 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:22:35.916 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:22:35.916 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_2 2023-12-10T19:22:35.916 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=deleted 2023-12-10T19:22:35.916 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:22:35.916 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:22:35.916 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:22:35.916 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_2 2023-12-10T19:22:35.917 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:22:35.917 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:22:35.917 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_2 2023-12-10T19:22:35.917 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_2 2023-12-10T19:22:35.917 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:22:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_2: (2) No such file or directory 2023-12-10T19:22:35.970 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image_id= 2023-12-10T19:22:35.970 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.1 2023-12-10T19:22:36.072 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_2 deleted 2023-12-10T19:22:36.072 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:22:36.072 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:22:36.072 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_2 2023-12-10T19:22:36.072 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=deleted 2023-12-10T19:22:36.072 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:22:36.072 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:22:36.072 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:22:36.073 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_2 2023-12-10T19:22:36.073 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:22:36.073 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:22:36.073 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_2 2023-12-10T19:22:36.073 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_2 2023-12-10T19:22:36.073 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:22:36.120 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_2: (2) No such file or directory 2023-12-10T19:22:36.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:22:36.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:22:36.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test deleted = deleted 2023-12-10T19:22:36.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:22:36.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ purge_snapshots cluster2 mirror image_2 2023-12-10T19:22:36.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:22:36.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:22:36.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_2 2023-12-10T19:22:36.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap purge mirror/image_2 2023-12-10T19:22:36.176 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image_retry cluster2 mirror image_2 2023-12-10T19:22:36.177 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:22:36.177 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:22:36.177 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_2 2023-12-10T19:22:36.177 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:22:36.177 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_2 2023-12-10T19:22:36.177 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:22:36.178 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:22:36.178 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_2 2023-12-10T19:22:36.178 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2023-12-10T19:22:36.226 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2023-12-10T19:22:36.272 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 0% complete...failed. 2023-12-10T19:22:36.272 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error: image still has watchers 2023-12-10T19:22:36.272 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-10T19:22:36.275 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:22:37.277 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_2 2023-12-10T19:22:37.277 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:22:37.277 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:22:37.277 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_2 2023-12-10T19:22:37.277 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2023-12-10T19:22:37.336 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2023-12-10T19:22:37.384 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 0% complete...failed. 2023-12-10T19:22:37.385 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error: image still has watchers 2023-12-10T19:22:37.385 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-10T19:22:37.389 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 2 2023-12-10T19:22:39.390 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_2 2023-12-10T19:22:39.390 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:22:39.390 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:22:39.390 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_2 2023-12-10T19:22:39.391 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2023-12-10T19:22:39.448 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2023-12-10T19:22:39.496 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 0% complete...failed. 2023-12-10T19:22:39.496 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error: image still has watchers 2023-12-10T19:22:39.496 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-10T19:22:39.499 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 4 2023-12-10T19:22:43.501 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_2 2023-12-10T19:22:43.501 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:22:43.501 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:22:43.501 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_2 2023-12-10T19:22:43.501 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2023-12-10T19:22:43.559 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2023-12-10T19:22:43.608 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 0% complete...failed. 2023-12-10T19:22:43.608 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error: image still has watchers 2023-12-10T19:22:43.608 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-10T19:22:43.612 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 8 2023-12-10T19:22:51.613 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_2 2023-12-10T19:22:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:22:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:22:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_2 2023-12-10T19:22:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2023-12-10T19:22:51.672 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2023-12-10T19:22:51.721 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 0% complete...failed. 2023-12-10T19:22:51.721 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error: image still has watchers 2023-12-10T19:22:51.721 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-10T19:22:51.725 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 16 2023-12-10T19:23:07.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_2 2023-12-10T19:23:07.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:23:07.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:23:07.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_2 2023-12-10T19:23:07.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2023-12-10T19:23:07.783 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2023-12-10T19:23:08.066 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-10T19:23:08.069 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:23:08.070 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_3 2023-12-10T19:23:08.070 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image cluster2 mirror image_3 512M 2023-12-10T19:23:08.070 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:23:08.070 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:23:08.070 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:23:08.070 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:23:08.070 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_3 2023-12-10T19:23:08.070 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:23:08.070 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local size=128 2023-12-10T19:23:08.070 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n 512M ] 2023-12-10T19:23:08.070 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ size=512M 2023-12-10T19:23:08.070 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:23:08.070 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/image_3 2023-12-10T19:23:08.117 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ enable_mirror cluster2 mirror image_3 2023-12-10T19:23:08.117 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:23:08.117 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:23:08.117 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_3 2023-12-10T19:23:08.117 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:23:08.117 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 snapshot 2023-12-10T19:23:08.912 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirroring enabled 2023-12-10T19:23:08.923 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2023-12-10T19:23:08.970 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_3': 2023-12-10T19:23:08.970 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 512 MiB in 128 objects 2023-12-10T19:23:08.970 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:23:08.971 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 1 2023-12-10T19:23:08.971 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 1960bb4e44aa 2023-12-10T19:23:08.971 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.1960bb4e44aa 2023-12-10T19:23:08.971 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:23:08.971 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering 2023-12-10T19:23:08.971 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:23:08.971 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:23:08.971 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:23:08 2023 2023-12-10T19:23:08.971 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:23:08 2023 2023-12-10T19:23:08.971 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:23:08 2023 2023-12-10T19:23:08.971 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:23:08.971 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:23:08.972 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: 0ad67c1c-3283-4167-ad76-759761f7e40f 2023-12-10T19:23:08.972 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: true 2023-12-10T19:23:08.974 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stress_write_image cluster2 mirror image_3 2023-12-10T19:23:08.974 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:23:08.974 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:23:08.975 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_3 2023-12-10T19:23:08.975 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-10T19:23:08.976 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local duration=12 2023-12-10T19:23:08.976 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T19:23:08.976 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2023-12-10T19:23:09.185 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:100 IOs, 38093824 bytes 2023-12-10T19:23:09.406 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:200 IOs, 99782656 bytes 2023-12-10T19:23:09.590 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:300 IOs, 150313984 bytes 2023-12-10T19:23:09.765 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:400 IOs, 174848000 bytes 2023-12-10T19:23:09.944 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:500 IOs, 211601408 bytes 2023-12-10T19:23:10.155 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:600 IOs, 260456448 bytes 2023-12-10T19:23:10.369 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:700 IOs, 320937984 bytes 2023-12-10T19:23:10.600 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:800 IOs, 391032832 bytes 2023-12-10T19:23:11.016 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:900 IOs, 436579328 bytes 2023-12-10T19:23:11.233 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1000 IOs, 479522816 bytes 2023-12-10T19:23:11.425 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1100 IOs, 518670336 bytes 2023-12-10T19:23:11.580 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1200 IOs, 542912512 bytes 2023-12-10T19:23:11.787 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1300 IOs, 587134976 bytes 2023-12-10T19:23:11.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1400 IOs, 626430976 bytes 2023-12-10T19:23:12.224 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1500 IOs, 688989184 bytes 2023-12-10T19:23:12.327 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1600 IOs, 689584128 bytes 2023-12-10T19:23:12.528 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1700 IOs, 730545152 bytes 2023-12-10T19:23:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1800 IOs, 794479616 bytes 2023-12-10T19:23:12.893 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.3532128051641046) > weight (0.25) 2023-12-10T19:23:12.893 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.2584670134406599) > weight (0.25) 2023-12-10T19:23:12.893 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.6583484459385853) > weight (0.25) 2023-12-10T19:23:12.893 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2023-12-10T19:23:12.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2023-12-10T19:23:12.894 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2023-12-10T19:23:12.895 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi076.stderr:2023-12-10T19:23:12.896+0000 7f890ce4b640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 53147) UID: 1000 2023-12-10T19:23:13.064 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1900 IOs, 878224384 bytes 2023-12-10T19:23:13.302 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2000 IOs, 955428864 bytes 2023-12-10T19:23:13.554 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2100 IOs, 1016461312 bytes 2023-12-10T19:23:13.743 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2200 IOs, 1065874432 bytes 2023-12-10T19:23:14.002 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2300 IOs, 1136823296 bytes 2023-12-10T19:23:14.176 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2400 IOs, 1169525760 bytes 2023-12-10T19:23:14.392 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2500 IOs, 1217681408 bytes 2023-12-10T19:23:14.589 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2600 IOs, 1260483584 bytes 2023-12-10T19:23:14.733 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2700 IOs, 1275122688 bytes 2023-12-10T19:23:14.893 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2800 IOs, 1301809152 bytes 2023-12-10T19:23:15.141 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2900 IOs, 1373531136 bytes 2023-12-10T19:23:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3000 IOs, 1403741184 bytes 2023-12-10T19:23:15.462 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3100 IOs, 1420831744 bytes 2023-12-10T19:23:15.565 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3200 IOs, 1423919104 bytes 2023-12-10T19:23:15.829 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3300 IOs, 1508645888 bytes 2023-12-10T19:23:15.958 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3400 IOs, 1509256192 bytes 2023-12-10T19:23:16.198 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3500 IOs, 1568721920 bytes 2023-12-10T19:23:16.407 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3600 IOs, 1622852608 bytes 2023-12-10T19:23:16.587 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3700 IOs, 1660372992 bytes 2023-12-10T19:23:16.735 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3800 IOs, 1677320192 bytes 2023-12-10T19:23:16.945 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3900 IOs, 1741362176 bytes 2023-12-10T19:23:17.133 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4000 IOs, 1768391680 bytes 2023-12-10T19:23:17.330 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4100 IOs, 1815498752 bytes 2023-12-10T19:23:17.548 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4200 IOs, 1871404032 bytes 2023-12-10T19:23:17.696 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4300 IOs, 1893732352 bytes 2023-12-10T19:23:17.886 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4400 IOs, 1934748672 bytes 2023-12-10T19:23:18.010 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4500 IOs, 1945504768 bytes 2023-12-10T19:23:18.157 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4600 IOs, 1967304704 bytes 2023-12-10T19:23:18.290 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4700 IOs, 1975934976 bytes 2023-12-10T19:23:18.520 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4800 IOs, 2040984576 bytes 2023-12-10T19:23:18.699 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4900 IOs, 2047396864 bytes 2023-12-10T19:23:18.943 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5000 IOs, 2116538368 bytes 2023-12-10T19:23:19.261 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5100 IOs, 2210971648 bytes 2023-12-10T19:23:19.418 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5200 IOs, 2244614144 bytes 2023-12-10T19:23:19.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5300 IOs, 2306805760 bytes 2023-12-10T19:23:19.938 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5400 IOs, 2381436928 bytes 2023-12-10T19:23:20.167 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5500 IOs, 2447012864 bytes 2023-12-10T19:23:20.318 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5600 IOs, 2469819392 bytes 2023-12-10T19:23:20.474 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5700 IOs, 2493132800 bytes 2023-12-10T19:23:20.719 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5800 IOs, 2564464640 bytes 2023-12-10T19:23:20.941 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5900 IOs, 2623868928 bytes 2023-12-10T19:23:20.984 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=124 2023-12-10T19:23:20.985 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T19:23:20.985 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 124 -eq 124 ] 2023-12-10T19:23:20.985 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:23:20.985 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_present cluster1 mirror image_3 present 2023-12-10T19:23:20.985 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:23:20.985 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:23:20.985 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_3 2023-12-10T19:23:20.985 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=present 2023-12-10T19:23:20.985 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:23:20.985 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:23:20.985 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:23:20.985 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_3 2023-12-10T19:23:20.985 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:23:20.985 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:23:20.985 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_3 2023-12-10T19:23:20.986 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_3 2023-12-10T19:23:20.986 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:23:21.039 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image_id=128b38ff1c08 2023-12-10T19:23:21.039 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.1 2023-12-10T19:23:21.141 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_3 present 128b38ff1c08 2023-12-10T19:23:21.141 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:23:21.141 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:23:21.141 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_3 2023-12-10T19:23:21.141 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=present 2023-12-10T19:23:21.141 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id=128b38ff1c08 2023-12-10T19:23:21.141 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:23:21.141 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:23:21.141 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_3 2023-12-10T19:23:21.141 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:23:21.141 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:23:21.141 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_3 2023-12-10T19:23:21.142 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_3 2023-12-10T19:23:21.142 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:23:21.200 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id=128b38ff1c08 2023-12-10T19:23:21.200 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 128b38ff1c08 2023-12-10T19:23:21.200 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -z 128b38ff1c08 -o 128b38ff1c08 = 128b38ff1c08 2023-12-10T19:23:21.200 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=present 2023-12-10T19:23:21.200 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test present = present 2023-12-10T19:23:21.200 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:23:21.200 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ disable_mirror cluster2 mirror image_3 2023-12-10T19:23:21.200 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:23:21.200 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:23:21.200 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_3 2023-12-10T19:23:21.201 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_3 2023-12-10T19:23:21.895 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2023-12-10T19:23:21.895 INFO:teuthology.orchestra.run:waiting for 600 2023-12-10T19:23:21.895 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2023-12-10T19:23:21.895 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2023-12-10T19:23:21.896 DEBUG:teuthology.orchestra.run.smithi076:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2023-12-10T19:23:21.899 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2023-12-10T19:23:21.899 INFO:tasks.rbd_mirror_thrash:waiting for 92 secs before thrashing 2023-12-10T19:23:21.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi076.stderr:2023-12-10T19:23:21.948+0000 7f3eaba39640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55b33c624800 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=372 2023-12-10T19:23:21.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi076.stderr:2023-12-10T19:23:21.948+0000 7f3eaba39640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55b33c624800 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=376, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=f4a845a0-65d8-4baa-abd4-fd2893f9af83, primary_snap_id=174, last_copied_object_number=128, snap_seqs={372=18446744073709551614}] 2023-12-10T19:23:26.952 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirroring disabled 2023-12-10T19:23:26.966 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2023-12-10T19:23:26.966 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:23:26.966 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:23:26.966 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_3 2023-12-10T19:23:26.966 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=deleted 2023-12-10T19:23:26.966 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:23:26.966 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:23:26.966 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:23:26.966 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_3 2023-12-10T19:23:26.966 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:23:26.966 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:23:26.966 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_3 2023-12-10T19:23:26.967 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_3 2023-12-10T19:23:26.967 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:23:27.018 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_3: (2) No such file or directory 2023-12-10T19:23:27.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image_id= 2023-12-10T19:23:27.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.1 2023-12-10T19:23:27.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_3 deleted 2023-12-10T19:23:27.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:23:27.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:23:27.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_3 2023-12-10T19:23:27.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=deleted 2023-12-10T19:23:27.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:23:27.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:23:27.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:23:27.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_3 2023-12-10T19:23:27.125 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:23:27.125 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:23:27.125 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_3 2023-12-10T19:23:27.125 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_3 2023-12-10T19:23:27.125 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:23:27.179 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_3: (2) No such file or directory 2023-12-10T19:23:27.183 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:23:27.183 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:23:27.184 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test deleted = deleted 2023-12-10T19:23:27.184 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:23:27.184 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ purge_snapshots cluster2 mirror image_3 2023-12-10T19:23:27.184 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:23:27.184 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:23:27.184 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_3 2023-12-10T19:23:27.184 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap purge mirror/image_3 2023-12-10T19:23:27.236 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image_retry cluster2 mirror image_3 2023-12-10T19:23:27.236 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:23:27.236 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:23:27.236 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_3 2023-12-10T19:23:27.236 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:23:27.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_3 2023-12-10T19:23:27.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:23:27.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:23:27.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_3 2023-12-10T19:23:27.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2023-12-10T19:23:27.287 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2023-12-10T19:23:27.336 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 0% complete...failed. 2023-12-10T19:23:27.337 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error: image still has watchers 2023-12-10T19:23:27.337 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-10T19:23:27.340 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:23:28.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_3 2023-12-10T19:23:28.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:23:28.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:23:28.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_3 2023-12-10T19:23:28.341 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2023-12-10T19:23:28.398 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2023-12-10T19:23:28.447 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 0% complete...failed. 2023-12-10T19:23:28.447 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error: image still has watchers 2023-12-10T19:23:28.447 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-10T19:23:28.450 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 2 2023-12-10T19:23:30.451 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_3 2023-12-10T19:23:30.452 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:23:30.452 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:23:30.452 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_3 2023-12-10T19:23:30.452 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2023-12-10T19:23:30.509 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2023-12-10T19:23:30.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 0% complete...failed. 2023-12-10T19:23:30.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error: image still has watchers 2023-12-10T19:23:30.557 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-10T19:23:30.561 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 4 2023-12-10T19:23:33.887 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2023-12-10T19:23:33.888 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2023-12-10T19:23:33.888 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2023-12-10T19:23:33.889 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi076.stderr:2023-12-10T19:23:33.891+0000 7f7f958e4640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 51113) UID: 1000 2023-12-10T19:23:34.562 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_3 2023-12-10T19:23:34.562 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:23:34.562 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:23:34.562 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_3 2023-12-10T19:23:34.562 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2023-12-10T19:23:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2023-12-10T19:23:34.671 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 0% complete...failed. 2023-12-10T19:23:34.672 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error: image still has watchers 2023-12-10T19:23:34.672 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-10T19:23:34.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 8 2023-12-10T19:23:36.889 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2023-12-10T19:23:36.889 INFO:teuthology.orchestra.run:waiting for 600 2023-12-10T19:23:36.889 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2023-12-10T19:23:36.889 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-12-10T19:23:36.889 DEBUG:teuthology.orchestra.run.smithi076:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-12-10T19:23:36.892 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-12-10T19:23:36.892 INFO:tasks.rbd_mirror_thrash:waiting for 100 secs before thrashing 2023-12-10T19:23:42.676 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_3 2023-12-10T19:23:42.677 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:23:42.677 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:23:42.677 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_3 2023-12-10T19:23:42.677 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2023-12-10T19:23:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2023-12-10T19:23:42.994 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 0% complete...failed. 2023-12-10T19:23:42.995 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error: image still has watchers 2023-12-10T19:23:42.995 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-10T19:23:42.999 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 16 2023-12-10T19:23:59.001 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_3 2023-12-10T19:23:59.002 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:23:59.002 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:23:59.002 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_3 2023-12-10T19:23:59.002 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2023-12-10T19:23:59.058 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2023-12-10T19:23:59.572 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-10T19:23:59.577 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:23:59.577 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_4 2023-12-10T19:23:59.577 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image cluster2 mirror image_4 512M 2023-12-10T19:23:59.577 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:23:59.577 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:23:59.577 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:23:59.577 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:23:59.577 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_4 2023-12-10T19:23:59.577 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:23:59.577 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local size=128 2023-12-10T19:23:59.577 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n 512M ] 2023-12-10T19:23:59.577 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ size=512M 2023-12-10T19:23:59.578 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:23:59.578 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/image_4 2023-12-10T19:23:59.629 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ enable_mirror cluster2 mirror image_4 2023-12-10T19:23:59.630 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:23:59.630 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:23:59.630 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_4 2023-12-10T19:23:59.630 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:23:59.630 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 snapshot 2023-12-10T19:24:00.654 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirroring enabled 2023-12-10T19:24:00.666 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2023-12-10T19:24:00.713 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_4': 2023-12-10T19:24:00.713 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 512 MiB in 128 objects 2023-12-10T19:24:00.713 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:24:00.713 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 1 2023-12-10T19:24:00.713 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 19a6d8136b36 2023-12-10T19:24:00.713 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.19a6d8136b36 2023-12-10T19:24:00.713 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:24:00.714 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering 2023-12-10T19:24:00.714 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:24:00.714 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:24:00.714 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:23:59 2023 2023-12-10T19:24:00.714 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:23:59 2023 2023-12-10T19:24:00.714 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:23:59 2023 2023-12-10T19:24:00.714 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:24:00.714 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:24:00.714 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: c6742cac-c85a-4ddd-a44e-5686398b46b9 2023-12-10T19:24:00.714 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: true 2023-12-10T19:24:00.718 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stress_write_image cluster2 mirror image_4 2023-12-10T19:24:00.718 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:24:00.718 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:24:00.718 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_4 2023-12-10T19:24:00.718 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-10T19:24:00.719 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local duration=11 2023-12-10T19:24:00.719 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T19:24:00.719 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2023-12-10T19:24:00.964 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:100 IOs, 39828480 bytes 2023-12-10T19:24:01.077 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:200 IOs, 51323904 bytes 2023-12-10T19:24:01.262 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:300 IOs, 91325440 bytes 2023-12-10T19:24:01.469 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:400 IOs, 133650432 bytes 2023-12-10T19:24:01.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:500 IOs, 217965568 bytes 2023-12-10T19:24:01.901 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:600 IOs, 243274752 bytes 2023-12-10T19:24:02.036 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:700 IOs, 263099392 bytes 2023-12-10T19:24:02.238 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:800 IOs, 303787008 bytes 2023-12-10T19:24:02.400 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:900 IOs, 348012544 bytes 2023-12-10T19:24:02.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1000 IOs, 422424576 bytes 2023-12-10T19:24:02.792 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1100 IOs, 426175488 bytes 2023-12-10T19:24:02.951 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1200 IOs, 454577152 bytes 2023-12-10T19:24:03.208 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1300 IOs, 535813120 bytes 2023-12-10T19:24:03.471 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1400 IOs, 609810432 bytes 2023-12-10T19:24:03.577 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1500 IOs, 610446336 bytes 2023-12-10T19:24:03.806 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1600 IOs, 659961856 bytes 2023-12-10T19:24:04.062 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1700 IOs, 736475136 bytes 2023-12-10T19:24:04.176 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1800 IOs, 737087488 bytes 2023-12-10T19:24:04.395 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1900 IOs, 796987392 bytes 2023-12-10T19:24:04.631 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2000 IOs, 860034048 bytes 2023-12-10T19:24:04.818 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2100 IOs, 900798464 bytes 2023-12-10T19:24:05.099 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2200 IOs, 985806848 bytes 2023-12-10T19:24:05.265 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2300 IOs, 1005536256 bytes 2023-12-10T19:24:05.453 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2400 IOs, 1040190464 bytes 2023-12-10T19:24:05.566 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2500 IOs, 1042257920 bytes 2023-12-10T19:24:05.746 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2600 IOs, 1070068736 bytes 2023-12-10T19:24:05.954 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2700 IOs, 1111555072 bytes 2023-12-10T19:24:06.175 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2800 IOs, 1168410624 bytes 2023-12-10T19:24:06.325 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2900 IOs, 1188085760 bytes 2023-12-10T19:24:06.436 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3000 IOs, 1203735552 bytes 2023-12-10T19:24:06.664 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3100 IOs, 1253443584 bytes 2023-12-10T19:24:06.875 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3200 IOs, 1310597120 bytes 2023-12-10T19:24:07.176 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3300 IOs, 1407991808 bytes 2023-12-10T19:24:07.455 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3400 IOs, 1513142272 bytes 2023-12-10T19:24:07.639 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3500 IOs, 1538067456 bytes 2023-12-10T19:24:07.826 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3600 IOs, 1572568064 bytes 2023-12-10T19:24:07.967 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3700 IOs, 1588465664 bytes 2023-12-10T19:24:08.162 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3800 IOs, 1635430400 bytes 2023-12-10T19:24:08.451 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3900 IOs, 1718224896 bytes 2023-12-10T19:24:08.608 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4000 IOs, 1742732288 bytes 2023-12-10T19:24:08.758 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4100 IOs, 1764206592 bytes 2023-12-10T19:24:08.958 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4200 IOs, 1811204096 bytes 2023-12-10T19:24:09.113 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4300 IOs, 1842757632 bytes 2023-12-10T19:24:09.273 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4400 IOs, 1865654272 bytes 2023-12-10T19:24:09.430 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4500 IOs, 1898216448 bytes 2023-12-10T19:24:09.596 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4600 IOs, 1917967360 bytes 2023-12-10T19:24:09.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4700 IOs, 1964038144 bytes 2023-12-10T19:24:09.963 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4800 IOs, 1983319040 bytes 2023-12-10T19:24:10.095 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4900 IOs, 1997241344 bytes 2023-12-10T19:24:10.266 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5000 IOs, 2031232000 bytes 2023-12-10T19:24:10.388 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5100 IOs, 2041210880 bytes 2023-12-10T19:24:10.515 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5200 IOs, 2077392896 bytes 2023-12-10T19:24:10.800 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5300 IOs, 2135649280 bytes 2023-12-10T19:24:10.933 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5400 IOs, 2148219904 bytes 2023-12-10T19:24:11.159 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5500 IOs, 2211118080 bytes 2023-12-10T19:24:11.357 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5600 IOs, 2257500160 bytes 2023-12-10T19:24:11.502 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5700 IOs, 2277685248 bytes 2023-12-10T19:24:11.615 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5800 IOs, 2281546752 bytes 2023-12-10T19:24:11.723 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=124 2023-12-10T19:24:11.723 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T19:24:11.724 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 124 -eq 124 ] 2023-12-10T19:24:11.724 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:24:11.724 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_present cluster1 mirror image_4 present 2023-12-10T19:24:11.724 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:24:11.724 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:24:11.724 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_4 2023-12-10T19:24:11.724 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=present 2023-12-10T19:24:11.724 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:24:11.724 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:24:11.724 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:24:11.724 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_4 2023-12-10T19:24:11.724 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:24:11.724 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:24:11.724 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_4 2023-12-10T19:24:11.724 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_4 2023-12-10T19:24:11.725 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:24:11.778 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image_id=128bddd0bbf3 2023-12-10T19:24:11.778 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.1 2023-12-10T19:24:11.879 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_4 present 128bddd0bbf3 2023-12-10T19:24:11.880 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:24:11.880 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:24:11.880 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_4 2023-12-10T19:24:11.880 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=present 2023-12-10T19:24:11.880 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id=128bddd0bbf3 2023-12-10T19:24:11.880 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:24:11.880 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:24:11.880 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_4 2023-12-10T19:24:11.880 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:24:11.880 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:24:11.880 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_4 2023-12-10T19:24:11.880 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_4 2023-12-10T19:24:11.880 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:24:11.940 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id=128bddd0bbf3 2023-12-10T19:24:11.940 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 128bddd0bbf3 2023-12-10T19:24:11.940 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -z 128bddd0bbf3 -o 128bddd0bbf3 = 128bddd0bbf3 2023-12-10T19:24:11.941 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=present 2023-12-10T19:24:11.941 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test present = present 2023-12-10T19:24:11.941 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:24:11.941 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ disable_mirror cluster2 mirror image_4 2023-12-10T19:24:11.941 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:24:11.941 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:24:11.941 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_4 2023-12-10T19:24:11.941 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_4 2023-12-10T19:24:12.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi076.stderr:2023-12-10T19:24:12.954+0000 7f3eaba39640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55b33bf13c00 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=374 2023-12-10T19:24:12.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi076.stderr:2023-12-10T19:24:12.954+0000 7f3eaba39640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55b33bf13c00 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=378, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=f4a845a0-65d8-4baa-abd4-fd2893f9af83, primary_snap_id=176, last_copied_object_number=128, snap_seqs={374=18446744073709551614}] 2023-12-10T19:24:17.944 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirroring disabled 2023-12-10T19:24:17.959 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2023-12-10T19:24:17.959 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:24:17.959 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:24:17.959 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_4 2023-12-10T19:24:17.959 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=deleted 2023-12-10T19:24:17.959 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:24:17.959 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:24:17.959 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:24:17.959 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_4 2023-12-10T19:24:17.960 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:24:17.960 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:24:17.960 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_4 2023-12-10T19:24:17.960 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_4 2023-12-10T19:24:17.960 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:24:18.004 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_4: (2) No such file or directory 2023-12-10T19:24:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image_id= 2023-12-10T19:24:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.1 2023-12-10T19:24:18.110 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_4 deleted 2023-12-10T19:24:18.110 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:24:18.110 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:24:18.110 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_4 2023-12-10T19:24:18.110 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=deleted 2023-12-10T19:24:18.110 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:24:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:24:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:24:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_4 2023-12-10T19:24:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:24:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:24:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_4 2023-12-10T19:24:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_4 2023-12-10T19:24:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:24:18.155 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_4: (2) No such file or directory 2023-12-10T19:24:18.159 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:24:18.159 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:24:18.160 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test deleted = deleted 2023-12-10T19:24:18.160 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:24:18.160 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ purge_snapshots cluster2 mirror image_4 2023-12-10T19:24:18.160 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:24:18.160 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:24:18.160 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_4 2023-12-10T19:24:18.160 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap purge mirror/image_4 2023-12-10T19:24:18.209 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image_retry cluster2 mirror image_4 2023-12-10T19:24:18.210 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:24:18.210 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:24:18.210 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_4 2023-12-10T19:24:18.210 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:24:18.210 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_4 2023-12-10T19:24:18.210 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:24:18.211 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:24:18.211 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_4 2023-12-10T19:24:18.211 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2023-12-10T19:24:18.258 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2023-12-10T19:24:18.304 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 0% complete...failed. 2023-12-10T19:24:18.304 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error: image still has watchers 2023-12-10T19:24:18.304 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-10T19:24:18.307 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:24:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_4 2023-12-10T19:24:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:24:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:24:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_4 2023-12-10T19:24:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2023-12-10T19:24:19.369 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2023-12-10T19:24:19.417 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 0% complete...failed. 2023-12-10T19:24:19.417 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error: image still has watchers 2023-12-10T19:24:19.417 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-10T19:24:19.420 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 2 2023-12-10T19:24:21.421 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_4 2023-12-10T19:24:21.421 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:24:21.421 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:24:21.421 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_4 2023-12-10T19:24:21.421 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2023-12-10T19:24:21.479 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2023-12-10T19:24:21.526 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 0% complete...failed. 2023-12-10T19:24:21.527 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error: image still has watchers 2023-12-10T19:24:21.527 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-10T19:24:21.530 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 4 2023-12-10T19:24:25.531 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_4 2023-12-10T19:24:25.532 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:24:25.532 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:24:25.532 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_4 2023-12-10T19:24:25.532 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2023-12-10T19:24:25.589 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2023-12-10T19:24:25.637 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 0% complete...failed. 2023-12-10T19:24:25.637 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error: image still has watchers 2023-12-10T19:24:25.637 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-10T19:24:25.641 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 8 2023-12-10T19:24:33.642 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_4 2023-12-10T19:24:33.642 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:24:33.642 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:24:33.642 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_4 2023-12-10T19:24:33.642 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2023-12-10T19:24:33.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2023-12-10T19:24:33.745 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 0% complete...failed. 2023-12-10T19:24:33.745 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error: image still has watchers 2023-12-10T19:24:33.745 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-10T19:24:33.749 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 16 2023-12-10T19:24:49.750 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_4 2023-12-10T19:24:49.750 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:24:49.751 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:24:49.751 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_4 2023-12-10T19:24:49.751 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2023-12-10T19:24:49.807 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2023-12-10T19:24:50.101 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-10T19:24:50.106 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:24:50.106 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_5 2023-12-10T19:24:50.106 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image cluster2 mirror image_5 512M 2023-12-10T19:24:50.106 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:24:50.106 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:24:50.107 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:24:50.107 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:24:50.107 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_5 2023-12-10T19:24:50.107 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:24:50.107 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local size=128 2023-12-10T19:24:50.107 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n 512M ] 2023-12-10T19:24:50.107 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ size=512M 2023-12-10T19:24:50.107 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:24:50.107 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/image_5 2023-12-10T19:24:50.155 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ enable_mirror cluster2 mirror image_5 2023-12-10T19:24:50.155 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:24:50.155 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:24:50.155 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_5 2023-12-10T19:24:50.155 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:24:50.155 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 snapshot 2023-12-10T19:24:50.960 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirroring enabled 2023-12-10T19:24:50.976 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2023-12-10T19:24:51.023 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_5': 2023-12-10T19:24:51.023 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 512 MiB in 128 objects 2023-12-10T19:24:51.023 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:24:51.023 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 1 2023-12-10T19:24:51.023 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 19dbc603a3a 2023-12-10T19:24:51.023 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.19dbc603a3a 2023-12-10T19:24:51.023 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:24:51.023 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering 2023-12-10T19:24:51.023 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:24:51.023 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:24:51.023 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:24:50 2023 2023-12-10T19:24:51.024 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:24:50 2023 2023-12-10T19:24:51.024 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:24:50 2023 2023-12-10T19:24:51.024 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:24:51.024 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:24:51.024 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: 06e26537-406b-48fb-96a9-75ee5f7af95c 2023-12-10T19:24:51.024 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: true 2023-12-10T19:24:51.027 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stress_write_image cluster2 mirror image_5 2023-12-10T19:24:51.027 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:24:51.027 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:24:51.027 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_5 2023-12-10T19:24:51.027 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-10T19:24:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local duration=10 2023-12-10T19:24:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T19:24:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2023-12-10T19:24:51.258 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:100 IOs, 40359936 bytes 2023-12-10T19:24:51.403 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr:2023-12-10T19:24:51.405+0000 7ff7ab455640 -1 bdev(0x5560d58aee00 /var/lib/ceph/osd/cluster2-0/block) _aio_thread got r=-28 ((28) No space left on device) 2023-12-10T19:24:51.405 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr:./src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7ff7ab455640 time 2023-12-10T19:24:51.408675+0000 2023-12-10T19:24:51.405 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr:./src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-12-10T19:24:51.405 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: ceph version 19.0.0-130-g3d2db44f (3d2db44fa991bc355074aa9b7647f63b77201fef) reef (dev) 2023-12-10T19:24:51.405 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x5560d3674750] 2023-12-10T19:24:51.405 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: 2: (KernelDevice::_aio_thread()+0x9bc) [0x5560d3f5c7fc] 2023-12-10T19:24:51.405 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: 3: ceph-osd(+0x15b40e1) [0x5560d3f3f0e1] 2023-12-10T19:24:51.405 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: 4: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7ff7b929fb43] 2023-12-10T19:24:51.405 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: 5: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7ff7b9331a00] 2023-12-10T19:24:51.405 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr:*** Caught signal (Aborted) ** 2023-12-10T19:24:51.405 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: in thread 7ff7ab455640 thread_name:bstore_aio 2023-12-10T19:24:51.406 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr:2023-12-10T19:24:51.405+0000 7ff7ab455640 -1 ./src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7ff7ab455640 time 2023-12-10T19:24:51.408675+0000 2023-12-10T19:24:51.406 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr:./src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-12-10T19:24:51.406 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: 2023-12-10T19:24:51.406 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: ceph version 19.0.0-130-g3d2db44f (3d2db44fa991bc355074aa9b7647f63b77201fef) reef (dev) 2023-12-10T19:24:51.406 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x5560d3674750] 2023-12-10T19:24:51.406 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: 2: (KernelDevice::_aio_thread()+0x9bc) [0x5560d3f5c7fc] 2023-12-10T19:24:51.407 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: 3: ceph-osd(+0x15b40e1) [0x5560d3f3f0e1] 2023-12-10T19:24:51.407 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: 4: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7ff7b929fb43] 2023-12-10T19:24:51.407 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: 5: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7ff7b9331a00] 2023-12-10T19:24:51.407 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: 2023-12-10T19:24:51.407 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: ceph version 19.0.0-130-g3d2db44f (3d2db44fa991bc355074aa9b7647f63b77201fef) reef (dev) 2023-12-10T19:24:51.407 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7ff7b924d520] 2023-12-10T19:24:51.407 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: 2: pthread_kill() 2023-12-10T19:24:51.407 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: 3: raise() 2023-12-10T19:24:51.408 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: 4: abort() 2023-12-10T19:24:51.408 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x17b) [0x5560d3674805] 2023-12-10T19:24:51.408 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: 6: (KernelDevice::_aio_thread()+0x9bc) [0x5560d3f5c7fc] 2023-12-10T19:24:51.408 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: 7: ceph-osd(+0x15b40e1) [0x5560d3f3f0e1] 2023-12-10T19:24:51.408 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: 8: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7ff7b929fb43] 2023-12-10T19:24:51.408 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: 9: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7ff7b9331a00] 2023-12-10T19:24:51.408 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr:2023-12-10T19:24:51.409+0000 7ff7ab455640 -1 *** Caught signal (Aborted) ** 2023-12-10T19:24:51.408 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: in thread 7ff7ab455640 thread_name:bstore_aio 2023-12-10T19:24:51.408 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: 2023-12-10T19:24:51.408 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: ceph version 19.0.0-130-g3d2db44f (3d2db44fa991bc355074aa9b7647f63b77201fef) reef (dev) 2023-12-10T19:24:51.408 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7ff7b924d520] 2023-12-10T19:24:51.408 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: 2: pthread_kill() 2023-12-10T19:24:51.408 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: 3: raise() 2023-12-10T19:24:51.408 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: 4: abort() 2023-12-10T19:24:51.408 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x17b) [0x5560d3674805] 2023-12-10T19:24:51.409 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: 6: (KernelDevice::_aio_thread()+0x9bc) [0x5560d3f5c7fc] 2023-12-10T19:24:51.409 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: 7: ceph-osd(+0x15b40e1) [0x5560d3f3f0e1] 2023-12-10T19:24:51.409 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: 8: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7ff7b929fb43] 2023-12-10T19:24:51.409 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: 9: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7ff7b9331a00] 2023-12-10T19:24:51.409 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-10T19:24:51.409 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: 2023-12-10T19:24:51.423 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: -199> 2023-12-10T19:24:51.405+0000 7ff7ab455640 -1 bdev(0x5560d58aee00 /var/lib/ceph/osd/cluster2-0/block) _aio_thread got r=-28 ((28) No space left on device) 2023-12-10T19:24:51.423 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: -68> 2023-12-10T19:24:51.405+0000 7ff7ab455640 -1 ./src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7ff7ab455640 time 2023-12-10T19:24:51.408675+0000 2023-12-10T19:24:51.423 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr:./src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-12-10T19:24:51.423 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: 2023-12-10T19:24:51.423 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: ceph version 19.0.0-130-g3d2db44f (3d2db44fa991bc355074aa9b7647f63b77201fef) reef (dev) 2023-12-10T19:24:51.423 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x5560d3674750] 2023-12-10T19:24:51.423 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: 2: (KernelDevice::_aio_thread()+0x9bc) [0x5560d3f5c7fc] 2023-12-10T19:24:51.423 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: 3: ceph-osd(+0x15b40e1) [0x5560d3f3f0e1] 2023-12-10T19:24:51.423 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: 4: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7ff7b929fb43] 2023-12-10T19:24:51.424 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: 5: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7ff7b9331a00] 2023-12-10T19:24:51.424 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: 2023-12-10T19:24:51.424 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: 0> 2023-12-10T19:24:51.409+0000 7ff7ab455640 -1 *** Caught signal (Aborted) ** 2023-12-10T19:24:51.424 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: in thread 7ff7ab455640 thread_name:bstore_aio 2023-12-10T19:24:51.424 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: 2023-12-10T19:24:51.424 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: ceph version 19.0.0-130-g3d2db44f (3d2db44fa991bc355074aa9b7647f63b77201fef) reef (dev) 2023-12-10T19:24:51.424 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7ff7b924d520] 2023-12-10T19:24:51.424 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: 2: pthread_kill() 2023-12-10T19:24:51.424 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: 3: raise() 2023-12-10T19:24:51.424 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: 4: abort() 2023-12-10T19:24:51.424 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x17b) [0x5560d3674805] 2023-12-10T19:24:51.424 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: 6: (KernelDevice::_aio_thread()+0x9bc) [0x5560d3f5c7fc] 2023-12-10T19:24:51.424 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: 7: ceph-osd(+0x15b40e1) [0x5560d3f3f0e1] 2023-12-10T19:24:51.424 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: 8: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7ff7b929fb43] 2023-12-10T19:24:51.424 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: 9: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7ff7b9331a00] 2023-12-10T19:24:51.425 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-10T19:24:51.425 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: 2023-12-10T19:24:51.436 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: -461> 2023-12-10T19:24:51.405+0000 7ff7ab455640 -1 bdev(0x5560d58aee00 /var/lib/ceph/osd/cluster2-0/block) _aio_thread got r=-28 ((28) No space left on device) 2023-12-10T19:24:51.437 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: -330> 2023-12-10T19:24:51.405+0000 7ff7ab455640 -1 ./src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7ff7ab455640 time 2023-12-10T19:24:51.408675+0000 2023-12-10T19:24:51.437 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr:./src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-12-10T19:24:51.437 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: 2023-12-10T19:24:51.437 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: ceph version 19.0.0-130-g3d2db44f (3d2db44fa991bc355074aa9b7647f63b77201fef) reef (dev) 2023-12-10T19:24:51.437 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x5560d3674750] 2023-12-10T19:24:51.437 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: 2: (KernelDevice::_aio_thread()+0x9bc) [0x5560d3f5c7fc] 2023-12-10T19:24:51.437 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: 3: ceph-osd(+0x15b40e1) [0x5560d3f3f0e1] 2023-12-10T19:24:51.437 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: 4: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7ff7b929fb43] 2023-12-10T19:24:51.437 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: 5: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7ff7b9331a00] 2023-12-10T19:24:51.437 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: 2023-12-10T19:24:51.437 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: -262> 2023-12-10T19:24:51.409+0000 7ff7ab455640 -1 *** Caught signal (Aborted) ** 2023-12-10T19:24:51.437 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: in thread 7ff7ab455640 thread_name:bstore_aio 2023-12-10T19:24:51.437 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: 2023-12-10T19:24:51.437 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: ceph version 19.0.0-130-g3d2db44f (3d2db44fa991bc355074aa9b7647f63b77201fef) reef (dev) 2023-12-10T19:24:51.437 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7ff7b924d520] 2023-12-10T19:24:51.438 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: 2: pthread_kill() 2023-12-10T19:24:51.438 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: 3: raise() 2023-12-10T19:24:51.438 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: 4: abort() 2023-12-10T19:24:51.438 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x17b) [0x5560d3674805] 2023-12-10T19:24:51.438 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: 6: (KernelDevice::_aio_thread()+0x9bc) [0x5560d3f5c7fc] 2023-12-10T19:24:51.438 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: 7: ceph-osd(+0x15b40e1) [0x5560d3f3f0e1] 2023-12-10T19:24:51.438 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: 8: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7ff7b929fb43] 2023-12-10T19:24:51.438 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: 9: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7ff7b9331a00] 2023-12-10T19:24:51.438 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-10T19:24:51.438 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr: 2023-12-10T19:24:51.678 INFO:tasks.ceph.cluster2.osd.0.smithi076.stderr:daemon-helper: command failed with exit status 5 2023-12-10T19:24:53.033 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:200 IOs, 88601600 bytes 2023-12-10T19:24:53.157 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:300 IOs, 118715392 bytes 2023-12-10T19:24:53.301 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:400 IOs, 167446528 bytes 2023-12-10T19:24:53.479 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:500 IOs, 248702976 bytes 2023-12-10T19:24:53.586 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:600 IOs, 258438144 bytes 2023-12-10T19:24:53.677 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:700 IOs, 280225792 bytes 2023-12-10T19:24:53.808 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:800 IOs, 332984320 bytes 2023-12-10T19:24:53.900 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.8284521446989823) > weight (0.25) 2023-12-10T19:24:53.900 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.30087486965887555) > weight (0.25) 2023-12-10T19:24:53.901 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.8240851384737241) > weight (0.25) 2023-12-10T19:24:53.901 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.5124676469719018) > weight (0.25) 2023-12-10T19:24:53.901 INFO:tasks.rbd_mirror_thrash:waiting for 107 secs before thrashing 2023-12-10T19:24:54.052 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:900 IOs, 456921088 bytes 2023-12-10T19:24:54.294 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1000 IOs, 530790400 bytes 2023-12-10T19:24:54.418 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1100 IOs, 575708160 bytes 2023-12-10T19:24:54.541 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1200 IOs, 610328576 bytes 2023-12-10T19:24:54.651 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1300 IOs, 631704576 bytes 2023-12-10T19:24:54.744 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1400 IOs, 650976256 bytes 2023-12-10T19:24:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1500 IOs, 692630528 bytes 2023-12-10T19:24:55.037 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1600 IOs, 728587264 bytes 2023-12-10T19:24:55.062 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~0s 2023-12-10T19:24:55.159 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1700 IOs, 775981056 bytes 2023-12-10T19:24:55.312 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1800 IOs, 812150784 bytes 2023-12-10T19:24:55.431 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1900 IOs, 839312384 bytes 2023-12-10T19:24:55.602 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2000 IOs, 903311360 bytes 2023-12-10T19:24:55.696 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2100 IOs, 908438528 bytes 2023-12-10T19:24:55.822 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2200 IOs, 931629056 bytes 2023-12-10T19:24:55.991 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2300 IOs, 1007029248 bytes 2023-12-10T19:24:56.182 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2400 IOs, 1071760384 bytes 2023-12-10T19:24:56.293 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2500 IOs, 1093036032 bytes 2023-12-10T19:24:56.416 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2600 IOs, 1115482112 bytes 2023-12-10T19:24:56.623 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2700 IOs, 1193985024 bytes 2023-12-10T19:24:56.762 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2800 IOs, 1224914944 bytes 2023-12-10T19:24:56.961 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2900 IOs, 1305766912 bytes 2023-12-10T19:24:57.132 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3000 IOs, 1371205632 bytes 2023-12-10T19:24:57.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3100 IOs, 1387930624 bytes 2023-12-10T19:24:57.350 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3200 IOs, 1417035776 bytes 2023-12-10T19:24:57.505 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3300 IOs, 1468637184 bytes 2023-12-10T19:24:57.640 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3400 IOs, 1517064192 bytes 2023-12-10T19:24:57.734 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3500 IOs, 1553552384 bytes 2023-12-10T19:24:57.885 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3600 IOs, 1605636096 bytes 2023-12-10T19:24:58.014 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3700 IOs, 1647372288 bytes 2023-12-10T19:24:58.187 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3800 IOs, 1713953792 bytes 2023-12-10T19:24:58.320 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3900 IOs, 1750667264 bytes 2023-12-10T19:24:58.421 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4000 IOs, 1767290880 bytes 2023-12-10T19:24:58.528 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4100 IOs, 1789916160 bytes 2023-12-10T19:24:58.715 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4200 IOs, 1868519424 bytes 2023-12-10T19:24:58.838 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4300 IOs, 1896843264 bytes 2023-12-10T19:24:58.974 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4400 IOs, 1930900480 bytes 2023-12-10T19:24:59.083 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4500 IOs, 1949998080 bytes 2023-12-10T19:24:59.187 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4600 IOs, 1992331264 bytes 2023-12-10T19:24:59.314 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4700 IOs, 2008476672 bytes 2023-12-10T19:24:59.468 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4800 IOs, 2054710272 bytes 2023-12-10T19:24:59.628 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4900 IOs, 2098578432 bytes 2023-12-10T19:24:59.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5000 IOs, 2172344320 bytes 2023-12-10T19:24:59.902 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5100 IOs, 2191177728 bytes 2023-12-10T19:25:00.026 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5200 IOs, 2214784000 bytes 2023-12-10T19:25:00.217 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5300 IOs, 2310897664 bytes 2023-12-10T19:25:00.366 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5400 IOs, 2341288960 bytes 2023-12-10T19:25:00.464 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~5s 2023-12-10T19:25:00.516 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5500 IOs, 2393663488 bytes 2023-12-10T19:25:00.662 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5600 IOs, 2440108032 bytes 2023-12-10T19:25:00.776 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5700 IOs, 2463745024 bytes 2023-12-10T19:25:00.922 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5800 IOs, 2512396288 bytes 2023-12-10T19:25:01.033 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=124 2023-12-10T19:25:01.033 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T19:25:01.033 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 124 -eq 124 ] 2023-12-10T19:25:01.033 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:25:01.034 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_present cluster1 mirror image_5 present 2023-12-10T19:25:01.034 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:25:01.034 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:25:01.034 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_5 2023-12-10T19:25:01.034 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=present 2023-12-10T19:25:01.034 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:25:01.034 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:25:01.034 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:25:01.034 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_5 2023-12-10T19:25:01.034 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:25:01.034 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:25:01.034 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_5 2023-12-10T19:25:01.034 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_5 2023-12-10T19:25:01.034 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:25:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image_id=128b92580304 2023-12-10T19:25:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.1 2023-12-10T19:25:01.183 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_5 present 128b92580304 2023-12-10T19:25:01.183 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:25:01.183 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:25:01.183 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_5 2023-12-10T19:25:01.183 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=present 2023-12-10T19:25:01.183 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id=128b92580304 2023-12-10T19:25:01.183 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:25:01.183 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:25:01.184 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_5 2023-12-10T19:25:01.184 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:25:01.184 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:25:01.184 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_5 2023-12-10T19:25:01.184 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_5 2023-12-10T19:25:01.184 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:25:01.243 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id=128b92580304 2023-12-10T19:25:01.243 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 128b92580304 2023-12-10T19:25:01.243 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -z 128b92580304 -o 128b92580304 = 128b92580304 2023-12-10T19:25:01.243 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=present 2023-12-10T19:25:01.243 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test present = present 2023-12-10T19:25:01.243 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:25:01.243 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ disable_mirror cluster2 mirror image_5 2023-12-10T19:25:01.243 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:25:01.243 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:25:01.243 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_5 2023-12-10T19:25:01.243 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_5 2023-12-10T19:25:01.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi076.stderr:2023-12-10T19:25:01.944+0000 7f3eaba39640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55b33c625000 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=376 2023-12-10T19:25:01.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi076.stderr:2023-12-10T19:25:01.944+0000 7f3eaba39640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55b33c625000 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=380, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=f4a845a0-65d8-4baa-abd4-fd2893f9af83, primary_snap_id=178, last_copied_object_number=128, snap_seqs={376=18446744073709551614}] 2023-12-10T19:25:05.869 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~11s 2023-12-10T19:25:06.041 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr:2023-12-10T19:25:06.040+0000 7f2c5add4640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5655570d8340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-10T19:25:06.041 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr:2023-12-10T19:25:06.040+0000 7f2c60de0640 -1 rbd::mirror::ImageReplayer: 0x5655574bd400 [3/06e26537-406b-48fb-96a9-75ee5f7af95c] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-10T19:25:06.950 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirroring disabled 2023-12-10T19:25:06.964 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2023-12-10T19:25:06.964 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:25:06.964 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:25:06.964 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_5 2023-12-10T19:25:06.964 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=deleted 2023-12-10T19:25:06.965 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:25:06.965 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:25:06.965 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:25:06.965 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_5 2023-12-10T19:25:06.965 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:25:06.965 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:25:06.965 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_5 2023-12-10T19:25:06.965 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_5 2023-12-10T19:25:06.965 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:25:07.023 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_5: (2) No such file or directory 2023-12-10T19:25:07.027 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image_id= 2023-12-10T19:25:07.027 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.1 2023-12-10T19:25:07.129 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_5 deleted 2023-12-10T19:25:07.129 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:25:07.129 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:25:07.129 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_5 2023-12-10T19:25:07.129 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=deleted 2023-12-10T19:25:07.129 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:25:07.129 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:25:07.129 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:25:07.129 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_5 2023-12-10T19:25:07.129 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:25:07.129 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:25:07.129 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_5 2023-12-10T19:25:07.130 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_5 2023-12-10T19:25:07.130 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:25:07.208 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_5: (2) No such file or directory 2023-12-10T19:25:07.213 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:25:07.213 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:25:07.213 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test deleted = deleted 2023-12-10T19:25:07.213 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:25:07.213 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ purge_snapshots cluster2 mirror image_5 2023-12-10T19:25:07.213 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:25:07.213 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:25:07.213 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_5 2023-12-10T19:25:07.213 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap purge mirror/image_5 2023-12-10T19:25:07.266 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image_retry cluster2 mirror image_5 2023-12-10T19:25:07.266 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:25:07.266 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:25:07.266 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_5 2023-12-10T19:25:07.266 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:25:07.267 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_5 2023-12-10T19:25:07.267 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:25:07.267 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:25:07.267 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_5 2023-12-10T19:25:07.267 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2023-12-10T19:25:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2023-12-10T19:25:07.362 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 0% complete...failed. 2023-12-10T19:25:07.362 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error: image still has watchers 2023-12-10T19:25:07.362 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-10T19:25:07.365 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:25:08.366 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_5 2023-12-10T19:25:08.367 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:25:08.367 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:25:08.367 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_5 2023-12-10T19:25:08.367 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2023-12-10T19:25:08.423 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2023-12-10T19:25:08.470 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 0% complete...failed. 2023-12-10T19:25:08.470 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error: image still has watchers 2023-12-10T19:25:08.471 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-10T19:25:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 2 2023-12-10T19:25:10.476 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_5 2023-12-10T19:25:10.476 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:25:10.476 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:25:10.476 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_5 2023-12-10T19:25:10.476 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2023-12-10T19:25:10.531 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2023-12-10T19:25:10.578 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 0% complete...failed. 2023-12-10T19:25:10.578 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error: image still has watchers 2023-12-10T19:25:10.578 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-10T19:25:10.582 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 4 2023-12-10T19:25:11.272 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~16s 2023-12-10T19:25:14.583 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_5 2023-12-10T19:25:14.583 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:25:14.583 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:25:14.583 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_5 2023-12-10T19:25:14.583 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2023-12-10T19:25:14.641 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2023-12-10T19:25:14.689 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 0% complete...failed. 2023-12-10T19:25:14.689 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error: image still has watchers 2023-12-10T19:25:14.689 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-10T19:25:14.693 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 8 2023-12-10T19:25:16.676 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~22s 2023-12-10T19:25:16.893 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2023-12-10T19:25:16.894 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2023-12-10T19:25:16.894 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2023-12-10T19:25:16.895 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi076.stderr:2023-12-10T19:25:16.896+0000 7f53b73ba640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 74324) UID: 1000 2023-12-10T19:25:21.895 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2023-12-10T19:25:21.895 INFO:teuthology.orchestra.run:waiting for 600 2023-12-10T19:25:21.895 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2023-12-10T19:25:21.896 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-12-10T19:25:21.896 DEBUG:teuthology.orchestra.run.smithi076:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-12-10T19:25:21.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-12-10T19:25:21.899 INFO:tasks.rbd_mirror_thrash:waiting for 64 secs before thrashing 2023-12-10T19:25:22.079 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~27s 2023-12-10T19:25:22.694 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_5 2023-12-10T19:25:22.694 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:25:22.694 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:25:22.694 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_5 2023-12-10T19:25:22.694 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2023-12-10T19:25:22.751 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2023-12-10T19:25:22.798 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 0% complete...failed. 2023-12-10T19:25:22.798 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error: image still has watchers 2023-12-10T19:25:22.798 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-10T19:25:22.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 16 2023-12-10T19:25:27.482 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~32s 2023-12-10T19:25:32.886 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~38s 2023-12-10T19:25:38.290 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~43s 2023-12-10T19:25:38.803 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_5 2023-12-10T19:25:38.803 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:25:38.803 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:25:38.803 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_5 2023-12-10T19:25:38.803 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2023-12-10T19:25:38.864 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2023-12-10T19:25:39.094 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-10T19:25:39.098 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:25:39.098 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_6 2023-12-10T19:25:39.098 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image cluster2 mirror image_6 512M 2023-12-10T19:25:39.098 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:25:39.098 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:25:39.099 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:25:39.099 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:25:39.099 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_6 2023-12-10T19:25:39.099 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:25:39.099 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local size=128 2023-12-10T19:25:39.099 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n 512M ] 2023-12-10T19:25:39.099 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ size=512M 2023-12-10T19:25:39.099 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:25:39.099 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/image_6 2023-12-10T19:25:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ enable_mirror cluster2 mirror image_6 2023-12-10T19:25:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:25:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:25:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_6 2023-12-10T19:25:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:25:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 snapshot 2023-12-10T19:25:39.981 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirroring enabled 2023-12-10T19:25:39.992 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2023-12-10T19:25:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_6': 2023-12-10T19:25:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 512 MiB in 128 objects 2023-12-10T19:25:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:25:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 1 2023-12-10T19:25:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 1a1af9fb7422 2023-12-10T19:25:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.1a1af9fb7422 2023-12-10T19:25:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:25:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering 2023-12-10T19:25:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:25:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:25:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:25:39 2023 2023-12-10T19:25:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:25:39 2023 2023-12-10T19:25:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:25:39 2023 2023-12-10T19:25:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:25:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:25:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: 93c3640e-3950-4afe-9d73-98c20ffd97ea 2023-12-10T19:25:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: true 2023-12-10T19:25:40.045 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stress_write_image cluster2 mirror image_6 2023-12-10T19:25:40.045 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:25:40.045 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:25:40.045 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_6 2023-12-10T19:25:40.045 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-10T19:25:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local duration=11 2023-12-10T19:25:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T19:25:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2023-12-10T19:25:40.237 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:100 IOs, 56662016 bytes 2023-12-10T19:25:40.346 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:200 IOs, 77424640 bytes 2023-12-10T19:25:40.472 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:300 IOs, 108087296 bytes 2023-12-10T19:25:40.645 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:400 IOs, 173934592 bytes 2023-12-10T19:25:40.802 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:500 IOs, 244549632 bytes 2023-12-10T19:25:40.958 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:600 IOs, 309885952 bytes 2023-12-10T19:25:41.122 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:700 IOs, 346986496 bytes 2023-12-10T19:25:41.218 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:800 IOs, 378189824 bytes 2023-12-10T19:25:41.399 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:900 IOs, 435556352 bytes 2023-12-10T19:25:41.516 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1000 IOs, 480447488 bytes 2023-12-10T19:25:41.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1100 IOs, 531461120 bytes 2023-12-10T19:25:41.831 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1200 IOs, 581742592 bytes 2023-12-10T19:25:42.005 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1300 IOs, 597651456 bytes 2023-12-10T19:25:42.134 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1400 IOs, 632995840 bytes 2023-12-10T19:25:42.225 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1500 IOs, 657011712 bytes 2023-12-10T19:25:42.348 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1600 IOs, 667830272 bytes 2023-12-10T19:25:42.471 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1700 IOs, 700448768 bytes 2023-12-10T19:25:42.564 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1800 IOs, 711775232 bytes 2023-12-10T19:25:42.655 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1900 IOs, 713991168 bytes 2023-12-10T19:25:42.756 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2000 IOs, 725754880 bytes 2023-12-10T19:25:42.890 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2100 IOs, 749891584 bytes 2023-12-10T19:25:42.999 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2200 IOs, 766980096 bytes 2023-12-10T19:25:43.147 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2300 IOs, 846056448 bytes 2023-12-10T19:25:43.300 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2400 IOs, 870967296 bytes 2023-12-10T19:25:43.406 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2500 IOs, 900121600 bytes 2023-12-10T19:25:43.483 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2600 IOs, 900734976 bytes 2023-12-10T19:25:43.644 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2700 IOs, 980388864 bytes 2023-12-10T19:25:43.694 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~49s 2023-12-10T19:25:43.774 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2800 IOs, 1013875712 bytes 2023-12-10T19:25:43.872 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2900 IOs, 1040365568 bytes 2023-12-10T19:25:44.011 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3000 IOs, 1093438464 bytes 2023-12-10T19:25:44.119 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3100 IOs, 1114315776 bytes 2023-12-10T19:25:44.253 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3200 IOs, 1160346624 bytes 2023-12-10T19:25:44.462 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3300 IOs, 1251116032 bytes 2023-12-10T19:25:44.569 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3400 IOs, 1267765248 bytes 2023-12-10T19:25:44.655 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3500 IOs, 1280552960 bytes 2023-12-10T19:25:44.773 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3600 IOs, 1315789824 bytes 2023-12-10T19:25:44.872 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3700 IOs, 1339778048 bytes 2023-12-10T19:25:44.974 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3800 IOs, 1367518208 bytes 2023-12-10T19:25:45.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3900 IOs, 1427000320 bytes 2023-12-10T19:25:45.249 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4000 IOs, 1486476288 bytes 2023-12-10T19:25:45.407 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4100 IOs, 1545811968 bytes 2023-12-10T19:25:45.553 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4200 IOs, 1606548480 bytes 2023-12-10T19:25:45.681 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4300 IOs, 1650952192 bytes 2023-12-10T19:25:45.830 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4400 IOs, 1716484096 bytes 2023-12-10T19:25:46.014 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4500 IOs, 1768585216 bytes 2023-12-10T19:25:46.105 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4600 IOs, 1795013632 bytes 2023-12-10T19:25:46.290 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4700 IOs, 1849866240 bytes 2023-12-10T19:25:46.410 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4800 IOs, 1880776704 bytes 2023-12-10T19:25:46.494 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4900 IOs, 1881399296 bytes 2023-12-10T19:25:46.609 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5000 IOs, 1915635712 bytes 2023-12-10T19:25:46.725 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5100 IOs, 1953842176 bytes 2023-12-10T19:25:46.903 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5200 IOs, 2018755584 bytes 2023-12-10T19:25:47.028 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5300 IOs, 2051801088 bytes 2023-12-10T19:25:47.132 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5400 IOs, 2080464896 bytes 2023-12-10T19:25:47.276 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5500 IOs, 2146464768 bytes 2023-12-10T19:25:47.358 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5600 IOs, 2147066880 bytes 2023-12-10T19:25:47.490 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5700 IOs, 2197398528 bytes 2023-12-10T19:25:47.595 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5800 IOs, 2225646592 bytes 2023-12-10T19:25:47.723 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5900 IOs, 2258195456 bytes 2023-12-10T19:25:47.852 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6000 IOs, 2304490496 bytes 2023-12-10T19:25:47.973 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6100 IOs, 2340678656 bytes 2023-12-10T19:25:48.108 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6200 IOs, 2380139520 bytes 2023-12-10T19:25:48.223 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6300 IOs, 2399555584 bytes 2023-12-10T19:25:48.417 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6400 IOs, 2462965760 bytes 2023-12-10T19:25:48.627 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6500 IOs, 2551639040 bytes 2023-12-10T19:25:48.739 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6600 IOs, 2576934912 bytes 2023-12-10T19:25:48.904 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6700 IOs, 2638029824 bytes 2023-12-10T19:25:49.000 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6800 IOs, 2649139200 bytes 2023-12-10T19:25:49.098 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~54s 2023-12-10T19:25:49.130 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6900 IOs, 2687515648 bytes 2023-12-10T19:25:49.222 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:7000 IOs, 2694557696 bytes 2023-12-10T19:25:49.425 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:7100 IOs, 2778148864 bytes 2023-12-10T19:25:49.513 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:7200 IOs, 2785591296 bytes 2023-12-10T19:25:49.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:7300 IOs, 2831268864 bytes 2023-12-10T19:25:49.824 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:7400 IOs, 2870218752 bytes 2023-12-10T19:25:50.068 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:7500 IOs, 2981295104 bytes 2023-12-10T19:25:50.194 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:7600 IOs, 3008629760 bytes 2023-12-10T19:25:50.316 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:7700 IOs, 3028115456 bytes 2023-12-10T19:25:50.459 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:7800 IOs, 3058830336 bytes 2023-12-10T19:25:50.549 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:7900 IOs, 3063859200 bytes 2023-12-10T19:25:50.670 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:8000 IOs, 3094813696 bytes 2023-12-10T19:25:50.748 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:8100 IOs, 3095426048 bytes 2023-12-10T19:25:50.824 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:8200 IOs, 3096051712 bytes 2023-12-10T19:25:50.909 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:8300 IOs, 3097365504 bytes 2023-12-10T19:25:50.995 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:8400 IOs, 3099649024 bytes 2023-12-10T19:25:51.052 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=124 2023-12-10T19:25:51.052 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T19:25:51.052 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 124 -eq 124 ] 2023-12-10T19:25:51.052 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:25:51.052 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_present cluster1 mirror image_6 present 2023-12-10T19:25:51.052 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:25:51.052 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:25:51.052 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_6 2023-12-10T19:25:51.052 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=present 2023-12-10T19:25:51.052 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:25:51.052 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:25:51.052 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:25:51.052 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_6 2023-12-10T19:25:51.053 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:25:51.053 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:25:51.053 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_6 2023-12-10T19:25:51.053 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_6 2023-12-10T19:25:51.053 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:25:51.104 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image_id=128b7d0fd9f7 2023-12-10T19:25:51.104 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.1 2023-12-10T19:25:51.206 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_6 present 128b7d0fd9f7 2023-12-10T19:25:51.206 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:25:51.206 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:25:51.206 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_6 2023-12-10T19:25:51.206 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=present 2023-12-10T19:25:51.206 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id=128b7d0fd9f7 2023-12-10T19:25:51.206 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:25:51.206 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:25:51.206 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_6 2023-12-10T19:25:51.207 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:25:51.207 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:25:51.207 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_6 2023-12-10T19:25:51.207 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_6 2023-12-10T19:25:51.207 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:25:51.265 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id=128b7d0fd9f7 2023-12-10T19:25:51.265 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 128b7d0fd9f7 2023-12-10T19:25:51.265 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -z 128b7d0fd9f7 -o 128b7d0fd9f7 = 128b7d0fd9f7 2023-12-10T19:25:51.265 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=present 2023-12-10T19:25:51.265 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test present = present 2023-12-10T19:25:51.265 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:25:51.265 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ disable_mirror cluster2 mirror image_6 2023-12-10T19:25:51.265 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:25:51.265 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:25:51.265 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_6 2023-12-10T19:25:51.266 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_6 2023-12-10T19:25:51.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi076.stderr:2023-12-10T19:25:51.987+0000 7f3eac23a640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55b33c130800 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=378 2023-12-10T19:25:51.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi076.stderr:2023-12-10T19:25:51.987+0000 7f3eac23a640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55b33c130800 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=382, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=f4a845a0-65d8-4baa-abd4-fd2893f9af83, primary_snap_id=17a, last_copied_object_number=128, snap_seqs={378=18446744073709551614}] 2023-12-10T19:25:54.502 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~59s 2023-12-10T19:25:57.000 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirroring disabled 2023-12-10T19:25:57.015 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2023-12-10T19:25:57.015 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:25:57.015 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:25:57.015 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_6 2023-12-10T19:25:57.015 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=deleted 2023-12-10T19:25:57.015 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:25:57.015 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:25:57.015 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:25:57.015 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_6 2023-12-10T19:25:57.015 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:25:57.015 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:25:57.015 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_6 2023-12-10T19:25:57.016 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_6 2023-12-10T19:25:57.016 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:25:57.062 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_6: (2) No such file or directory 2023-12-10T19:25:57.067 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image_id= 2023-12-10T19:25:57.067 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.1 2023-12-10T19:25:57.170 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_6 deleted 2023-12-10T19:25:57.170 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:25:57.170 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:25:57.170 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_6 2023-12-10T19:25:57.171 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=deleted 2023-12-10T19:25:57.171 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:25:57.171 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:25:57.171 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:25:57.171 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_6 2023-12-10T19:25:57.171 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:25:57.171 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:25:57.171 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_6 2023-12-10T19:25:57.171 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_6 2023-12-10T19:25:57.171 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:25:57.216 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_6: (2) No such file or directory 2023-12-10T19:25:57.220 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:25:57.220 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:25:57.220 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test deleted = deleted 2023-12-10T19:25:57.220 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:25:57.220 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ purge_snapshots cluster2 mirror image_6 2023-12-10T19:25:57.220 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:25:57.220 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:25:57.220 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_6 2023-12-10T19:25:57.221 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap purge mirror/image_6 2023-12-10T19:25:57.270 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image_retry cluster2 mirror image_6 2023-12-10T19:25:57.271 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:25:57.271 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:25:57.271 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_6 2023-12-10T19:25:57.271 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:25:57.271 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_6 2023-12-10T19:25:57.271 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:25:57.272 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:25:57.272 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_6 2023-12-10T19:25:57.272 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2023-12-10T19:25:57.319 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2023-12-10T19:25:57.364 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 0% complete...failed. 2023-12-10T19:25:57.364 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error: image still has watchers 2023-12-10T19:25:57.364 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-10T19:25:57.367 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:25:58.368 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_6 2023-12-10T19:25:58.369 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:25:58.369 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:25:58.369 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_6 2023-12-10T19:25:58.369 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2023-12-10T19:25:58.427 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2023-12-10T19:25:58.476 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 0% complete...failed. 2023-12-10T19:25:58.476 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error: image still has watchers 2023-12-10T19:25:58.476 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-10T19:25:58.480 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 2 2023-12-10T19:25:59.905 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~65s 2023-12-10T19:26:00.481 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_6 2023-12-10T19:26:00.481 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:26:00.481 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:26:00.482 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_6 2023-12-10T19:26:00.482 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2023-12-10T19:26:00.539 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2023-12-10T19:26:00.586 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 0% complete...failed. 2023-12-10T19:26:00.586 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error: image still has watchers 2023-12-10T19:26:00.586 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-10T19:26:00.590 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 4 2023-12-10T19:26:04.591 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_6 2023-12-10T19:26:04.591 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:26:04.591 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:26:04.591 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_6 2023-12-10T19:26:04.591 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2023-12-10T19:26:04.650 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2023-12-10T19:26:04.696 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 0% complete...failed. 2023-12-10T19:26:04.696 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error: image still has watchers 2023-12-10T19:26:04.696 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-10T19:26:04.700 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 8 2023-12-10T19:26:05.309 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~70s 2023-12-10T19:26:10.713 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~76s 2023-12-10T19:26:12.701 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_6 2023-12-10T19:26:12.701 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:26:12.701 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:26:12.701 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_6 2023-12-10T19:26:12.701 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2023-12-10T19:26:12.757 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2023-12-10T19:26:12.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 0% complete...failed. 2023-12-10T19:26:12.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error: image still has watchers 2023-12-10T19:26:12.804 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-10T19:26:12.807 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 16 2023-12-10T19:26:16.116 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~81s 2023-12-10T19:26:21.520 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~86s 2023-12-10T19:26:25.900 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7823626356763032) > weight (0.25) 2023-12-10T19:26:25.900 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.5948027514604399) > weight (0.25) 2023-12-10T19:26:25.900 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2023-12-10T19:26:25.901 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2023-12-10T19:26:25.901 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2023-12-10T19:26:25.902 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi076.stderr:2023-12-10T19:26:25.906+0000 7f8349391640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 68167) UID: 1000 2023-12-10T19:26:26.924 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~92s 2023-12-10T19:26:28.808 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_6 2023-12-10T19:26:28.808 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:26:28.809 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:26:28.809 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_6 2023-12-10T19:26:28.809 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2023-12-10T19:26:28.864 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2023-12-10T19:26:29.087 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-10T19:26:29.091 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:26:29.091 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_7 2023-12-10T19:26:29.091 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image cluster2 mirror image_7 512M 2023-12-10T19:26:29.091 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:26:29.092 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:26:29.092 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:26:29.092 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:26:29.092 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_7 2023-12-10T19:26:29.092 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:26:29.092 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local size=128 2023-12-10T19:26:29.092 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n 512M ] 2023-12-10T19:26:29.092 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ size=512M 2023-12-10T19:26:29.092 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:26:29.092 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/image_7 2023-12-10T19:26:29.138 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ enable_mirror cluster2 mirror image_7 2023-12-10T19:26:29.138 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:26:29.138 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:26:29.138 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_7 2023-12-10T19:26:29.138 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:26:29.138 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 snapshot 2023-12-10T19:26:30.062 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirroring enabled 2023-12-10T19:26:30.073 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2023-12-10T19:26:30.121 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_7': 2023-12-10T19:26:30.121 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 512 MiB in 128 objects 2023-12-10T19:26:30.121 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:26:30.122 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 1 2023-12-10T19:26:30.122 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 1a50b3300c32 2023-12-10T19:26:30.122 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.1a50b3300c32 2023-12-10T19:26:30.122 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:26:30.122 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering 2023-12-10T19:26:30.122 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:26:30.122 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:26:30.122 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:26:29 2023 2023-12-10T19:26:30.122 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:26:29 2023 2023-12-10T19:26:30.122 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:26:29 2023 2023-12-10T19:26:30.122 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:26:30.122 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:26:30.122 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: 674cd4cc-59c5-4da3-b31d-222f182f5471 2023-12-10T19:26:30.122 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: true 2023-12-10T19:26:30.125 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stress_write_image cluster2 mirror image_7 2023-12-10T19:26:30.126 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:26:30.126 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:26:30.126 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_7 2023-12-10T19:26:30.126 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-10T19:26:30.127 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local duration=6 2023-12-10T19:26:30.127 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T19:26:30.127 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2023-12-10T19:26:30.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:100 IOs, 61948928 bytes 2023-12-10T19:26:30.504 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:200 IOs, 154448896 bytes 2023-12-10T19:26:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:300 IOs, 199569408 bytes 2023-12-10T19:26:30.741 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:400 IOs, 200183808 bytes 2023-12-10T19:26:30.850 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:500 IOs, 229789696 bytes 2023-12-10T19:26:30.936 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:600 IOs, 232172544 bytes 2023-12-10T19:26:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:700 IOs, 279003136 bytes 2023-12-10T19:26:31.168 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:800 IOs, 288370688 bytes 2023-12-10T19:26:31.301 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:900 IOs, 337720320 bytes 2023-12-10T19:26:31.442 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1000 IOs, 405123072 bytes 2023-12-10T19:26:31.613 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1100 IOs, 442309632 bytes 2023-12-10T19:26:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1200 IOs, 505809920 bytes 2023-12-10T19:26:31.881 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1300 IOs, 516370432 bytes 2023-12-10T19:26:31.992 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1400 IOs, 547435520 bytes 2023-12-10T19:26:32.130 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1500 IOs, 592703488 bytes 2023-12-10T19:26:32.303 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1600 IOs, 677736448 bytes 2023-12-10T19:26:32.328 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~97s 2023-12-10T19:26:32.439 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1700 IOs, 711988224 bytes 2023-12-10T19:26:32.520 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1800 IOs, 733094912 bytes 2023-12-10T19:26:32.721 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1900 IOs, 801657856 bytes 2023-12-10T19:26:32.861 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2000 IOs, 837716992 bytes 2023-12-10T19:26:33.017 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2100 IOs, 913320960 bytes 2023-12-10T19:26:33.221 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2200 IOs, 986455040 bytes 2023-12-10T19:26:33.319 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2300 IOs, 996761600 bytes 2023-12-10T19:26:33.483 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2400 IOs, 1057943552 bytes 2023-12-10T19:26:33.624 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2500 IOs, 1108097024 bytes 2023-12-10T19:26:33.776 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2600 IOs, 1176999936 bytes 2023-12-10T19:26:33.902 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2023-12-10T19:26:33.902 INFO:teuthology.orchestra.run:waiting for 600 2023-12-10T19:26:33.902 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2023-12-10T19:26:33.903 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2023-12-10T19:26:33.903 DEBUG:teuthology.orchestra.run.smithi076:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2023-12-10T19:26:33.905 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2023-12-10T19:26:33.905 INFO:tasks.rbd_mirror_thrash:waiting for 73 secs before thrashing 2023-12-10T19:26:33.923 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2700 IOs, 1211939840 bytes 2023-12-10T19:26:34.124 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2800 IOs, 1309499392 bytes 2023-12-10T19:26:34.246 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2900 IOs, 1318579200 bytes 2023-12-10T19:26:34.413 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3000 IOs, 1377967104 bytes 2023-12-10T19:26:34.542 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3100 IOs, 1418401792 bytes 2023-12-10T19:26:34.713 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3200 IOs, 1494068224 bytes 2023-12-10T19:26:34.809 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3300 IOs, 1497022464 bytes 2023-12-10T19:26:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3400 IOs, 1540981760 bytes 2023-12-10T19:26:35.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3500 IOs, 1566180352 bytes 2023-12-10T19:26:35.225 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3600 IOs, 1617776640 bytes 2023-12-10T19:26:35.357 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3700 IOs, 1681752064 bytes 2023-12-10T19:26:35.684 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3800 IOs, 1762683904 bytes 2023-12-10T19:26:35.861 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3900 IOs, 1819348992 bytes 2023-12-10T19:26:36.006 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4000 IOs, 1869842432 bytes 2023-12-10T19:26:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=124 2023-12-10T19:26:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T19:26:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 124 -eq 124 ] 2023-12-10T19:26:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:26:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_present cluster1 mirror image_7 present 2023-12-10T19:26:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:26:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:26:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_7 2023-12-10T19:26:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=present 2023-12-10T19:26:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:26:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:26:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:26:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_7 2023-12-10T19:26:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:26:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:26:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_7 2023-12-10T19:26:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_7 2023-12-10T19:26:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:26:36.187 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image_id=128b63200116 2023-12-10T19:26:36.187 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.1 2023-12-10T19:26:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_7 present 128b63200116 2023-12-10T19:26:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:26:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:26:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_7 2023-12-10T19:26:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=present 2023-12-10T19:26:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id=128b63200116 2023-12-10T19:26:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:26:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:26:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_7 2023-12-10T19:26:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:26:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:26:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_7 2023-12-10T19:26:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_7 2023-12-10T19:26:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:26:36.348 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id=128b63200116 2023-12-10T19:26:36.349 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 128b63200116 2023-12-10T19:26:36.349 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -z 128b63200116 -o 128b63200116 = 128b63200116 2023-12-10T19:26:36.349 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=present 2023-12-10T19:26:36.349 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test present = present 2023-12-10T19:26:36.349 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:26:36.349 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ disable_mirror cluster2 mirror image_7 2023-12-10T19:26:36.349 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:26:36.349 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:26:36.349 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_7 2023-12-10T19:26:36.349 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_7 2023-12-10T19:26:37.046 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr:2023-12-10T19:26:37.049+0000 7f2c5add4640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x565556fcb520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-10T19:26:37.046 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr:2023-12-10T19:26:37.049+0000 7f2c615e1640 -1 rbd::mirror::ImageReplayer: 0x565556ff0500 [3/674cd4cc-59c5-4da3-b31d-222f182f5471] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-10T19:26:37.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi076.stderr:2023-12-10T19:26:37.097+0000 7f3eac23a640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55b33c131800 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=380 2023-12-10T19:26:37.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi076.stderr:2023-12-10T19:26:37.097+0000 7f3eac23a640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55b33c131800 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=384, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=f4a845a0-65d8-4baa-abd4-fd2893f9af83, primary_snap_id=17c, last_copied_object_number=128, snap_seqs={380=18446744073709551614}] 2023-12-10T19:26:37.732 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~103s 2023-12-10T19:26:40.901 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.3394127811167702) > weight (0.25) 2023-12-10T19:26:40.902 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.9944581620610109) > weight (0.25) 2023-12-10T19:26:40.902 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.687399794598428) > weight (0.25) 2023-12-10T19:26:40.902 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.2616797035791648) > weight (0.25) 2023-12-10T19:26:40.902 INFO:tasks.rbd_mirror_thrash:waiting for 86 secs before thrashing 2023-12-10T19:26:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirroring disabled 2023-12-10T19:26:42.128 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2023-12-10T19:26:42.128 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:26:42.128 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:26:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_7 2023-12-10T19:26:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=deleted 2023-12-10T19:26:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:26:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:26:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:26:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_7 2023-12-10T19:26:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:26:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:26:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_7 2023-12-10T19:26:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_7 2023-12-10T19:26:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:26:42.175 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_7: (2) No such file or directory 2023-12-10T19:26:42.179 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image_id= 2023-12-10T19:26:42.179 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.1 2023-12-10T19:26:42.281 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_7 deleted 2023-12-10T19:26:42.281 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:26:42.281 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:26:42.281 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_7 2023-12-10T19:26:42.281 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=deleted 2023-12-10T19:26:42.281 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:26:42.281 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:26:42.281 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:26:42.281 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_7 2023-12-10T19:26:42.282 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:26:42.282 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:26:42.282 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_7 2023-12-10T19:26:42.282 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_7 2023-12-10T19:26:42.282 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:26:42.327 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_7: (2) No such file or directory 2023-12-10T19:26:42.331 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:26:42.331 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:26:42.331 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test deleted = deleted 2023-12-10T19:26:42.331 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:26:42.331 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ purge_snapshots cluster2 mirror image_7 2023-12-10T19:26:42.331 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:26:42.331 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:26:42.332 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_7 2023-12-10T19:26:42.332 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap purge mirror/image_7 2023-12-10T19:26:42.380 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image_retry cluster2 mirror image_7 2023-12-10T19:26:42.380 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:26:42.380 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:26:42.380 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_7 2023-12-10T19:26:42.381 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:26:42.381 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_7 2023-12-10T19:26:42.382 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:26:42.382 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:26:42.382 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_7 2023-12-10T19:26:42.382 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2023-12-10T19:26:42.428 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2023-12-10T19:26:42.474 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 0% complete...failed. 2023-12-10T19:26:42.474 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error: image still has watchers 2023-12-10T19:26:42.474 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-10T19:26:42.477 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:26:43.137 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~108s 2023-12-10T19:26:43.478 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_7 2023-12-10T19:26:43.478 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:26:43.479 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:26:43.479 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_7 2023-12-10T19:26:43.479 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2023-12-10T19:26:43.533 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2023-12-10T19:26:43.584 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 0% complete...failed. 2023-12-10T19:26:43.584 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error: image still has watchers 2023-12-10T19:26:43.585 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-10T19:26:43.588 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 2 2023-12-10T19:26:45.590 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_7 2023-12-10T19:26:45.590 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:26:45.590 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:26:45.590 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_7 2023-12-10T19:26:45.590 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2023-12-10T19:26:45.649 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2023-12-10T19:26:45.696 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 0% complete...failed. 2023-12-10T19:26:45.696 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error: image still has watchers 2023-12-10T19:26:45.696 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-10T19:26:45.700 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 4 2023-12-10T19:26:48.541 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~113s 2023-12-10T19:26:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_7 2023-12-10T19:26:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:26:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:26:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_7 2023-12-10T19:26:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2023-12-10T19:26:49.760 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2023-12-10T19:26:49.806 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 0% complete...failed. 2023-12-10T19:26:49.806 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error: image still has watchers 2023-12-10T19:26:49.806 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-10T19:26:49.810 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 8 2023-12-10T19:26:53.945 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~119s 2023-12-10T19:26:57.811 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_7 2023-12-10T19:26:57.811 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:26:57.811 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:26:57.811 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_7 2023-12-10T19:26:57.811 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2023-12-10T19:26:57.872 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2023-12-10T19:26:57.919 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 0% complete...failed. 2023-12-10T19:26:57.919 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error: image still has watchers 2023-12-10T19:26:57.919 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-10T19:26:57.923 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 16 2023-12-10T19:26:59.349 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~124s 2023-12-10T19:27:04.753 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~130s 2023-12-10T19:27:10.157 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~135s 2023-12-10T19:27:13.924 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_7 2023-12-10T19:27:13.924 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:27:13.925 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:27:13.925 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_7 2023-12-10T19:27:13.925 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2023-12-10T19:27:13.982 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2023-12-10T19:27:14.212 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-10T19:27:14.216 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:27:14.216 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_8 2023-12-10T19:27:14.217 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image cluster2 mirror image_8 512M 2023-12-10T19:27:14.217 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:27:14.217 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:27:14.217 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:27:14.217 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:27:14.217 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_8 2023-12-10T19:27:14.217 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:27:14.217 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local size=128 2023-12-10T19:27:14.217 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n 512M ] 2023-12-10T19:27:14.218 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ size=512M 2023-12-10T19:27:14.218 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:27:14.218 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/image_8 2023-12-10T19:27:14.266 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ enable_mirror cluster2 mirror image_8 2023-12-10T19:27:14.266 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:27:14.266 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:27:14.266 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_8 2023-12-10T19:27:14.266 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:27:14.266 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 snapshot 2023-12-10T19:27:15.074 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirroring enabled 2023-12-10T19:27:15.087 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 info mirror/image_8 2023-12-10T19:27:15.135 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_8': 2023-12-10T19:27:15.135 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 512 MiB in 128 objects 2023-12-10T19:27:15.135 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:27:15.136 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 1 2023-12-10T19:27:15.136 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 1a8f1a6c894f 2023-12-10T19:27:15.136 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.1a8f1a6c894f 2023-12-10T19:27:15.136 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:27:15.136 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering 2023-12-10T19:27:15.136 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:27:15.136 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:27:15.136 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:27:14 2023 2023-12-10T19:27:15.137 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:27:14 2023 2023-12-10T19:27:15.137 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:27:14 2023 2023-12-10T19:27:15.137 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:27:15.137 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:27:15.137 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: 42acbbfc-5f58-4d6d-a951-356e6e77b208 2023-12-10T19:27:15.137 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: true 2023-12-10T19:27:15.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stress_write_image cluster2 mirror image_8 2023-12-10T19:27:15.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:27:15.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:27:15.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_8 2023-12-10T19:27:15.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-10T19:27:15.141 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local duration=11 2023-12-10T19:27:15.141 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T19:27:15.142 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2023-12-10T19:27:15.536 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:100 IOs, 20051968 bytes 2023-12-10T19:27:15.561 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~140s 2023-12-10T19:27:15.668 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:200 IOs, 65378304 bytes 2023-12-10T19:27:15.880 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:300 IOs, 184991744 bytes 2023-12-10T19:27:15.989 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:400 IOs, 205412352 bytes 2023-12-10T19:27:16.105 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:500 IOs, 235547648 bytes 2023-12-10T19:27:16.224 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:600 IOs, 268464128 bytes 2023-12-10T19:27:16.354 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:700 IOs, 306081792 bytes 2023-12-10T19:27:16.471 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:800 IOs, 337475584 bytes 2023-12-10T19:27:16.645 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:900 IOs, 455212032 bytes 2023-12-10T19:27:16.754 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1000 IOs, 468444160 bytes 2023-12-10T19:27:16.899 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1100 IOs, 511015936 bytes 2023-12-10T19:27:16.983 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1200 IOs, 520624128 bytes 2023-12-10T19:27:17.103 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1300 IOs, 551413760 bytes 2023-12-10T19:27:17.259 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1400 IOs, 610705408 bytes 2023-12-10T19:27:17.350 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1500 IOs, 621635584 bytes 2023-12-10T19:27:17.437 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1600 IOs, 629519360 bytes 2023-12-10T19:27:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1700 IOs, 649772032 bytes 2023-12-10T19:27:17.754 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1800 IOs, 797563904 bytes 2023-12-10T19:27:17.895 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1900 IOs, 828143616 bytes 2023-12-10T19:27:18.098 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2000 IOs, 922708992 bytes 2023-12-10T19:27:18.194 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2100 IOs, 943156224 bytes 2023-12-10T19:27:18.279 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2200 IOs, 948283392 bytes 2023-12-10T19:27:18.453 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2300 IOs, 1006437376 bytes 2023-12-10T19:27:18.556 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2400 IOs, 1030304768 bytes 2023-12-10T19:27:18.659 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2500 IOs, 1061745664 bytes 2023-12-10T19:27:18.797 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2600 IOs, 1119126528 bytes 2023-12-10T19:27:18.913 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2700 IOs, 1142347776 bytes 2023-12-10T19:27:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2800 IOs, 1197577216 bytes 2023-12-10T19:27:19.173 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2900 IOs, 1225617408 bytes 2023-12-10T19:27:19.255 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3000 IOs, 1226231808 bytes 2023-12-10T19:27:19.382 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3100 IOs, 1275578368 bytes 2023-12-10T19:27:19.486 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3200 IOs, 1293920256 bytes 2023-12-10T19:27:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3300 IOs, 1337878528 bytes 2023-12-10T19:27:19.777 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3400 IOs, 1402827776 bytes 2023-12-10T19:27:19.887 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3500 IOs, 1432521728 bytes 2023-12-10T19:27:20.000 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3600 IOs, 1449426944 bytes 2023-12-10T19:27:20.147 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3700 IOs, 1508401152 bytes 2023-12-10T19:27:20.303 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3800 IOs, 1556271104 bytes 2023-12-10T19:27:20.452 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3900 IOs, 1605320704 bytes 2023-12-10T19:27:20.576 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4000 IOs, 1645744128 bytes 2023-12-10T19:27:20.704 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4100 IOs, 1678527488 bytes 2023-12-10T19:27:20.855 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4200 IOs, 1719354368 bytes 2023-12-10T19:27:20.964 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~146s 2023-12-10T19:27:20.998 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4300 IOs, 1769137152 bytes 2023-12-10T19:27:21.076 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4400 IOs, 1769757696 bytes 2023-12-10T19:27:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4500 IOs, 1799892992 bytes 2023-12-10T19:27:21.305 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4600 IOs, 1842689024 bytes 2023-12-10T19:27:21.442 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4700 IOs, 1860560896 bytes 2023-12-10T19:27:21.533 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4800 IOs, 1874497536 bytes 2023-12-10T19:27:21.640 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4900 IOs, 1897260032 bytes 2023-12-10T19:27:21.769 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5000 IOs, 1946189824 bytes 2023-12-10T19:27:21.871 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5100 IOs, 1968696320 bytes 2023-12-10T19:27:21.988 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5200 IOs, 1995185152 bytes 2023-12-10T19:27:22.119 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5300 IOs, 2036265984 bytes 2023-12-10T19:27:22.257 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5400 IOs, 2097916928 bytes 2023-12-10T19:27:22.458 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5500 IOs, 2160508928 bytes 2023-12-10T19:27:22.595 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5600 IOs, 2205927424 bytes 2023-12-10T19:27:22.692 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5700 IOs, 2218167296 bytes 2023-12-10T19:27:22.861 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5800 IOs, 2279284736 bytes 2023-12-10T19:27:23.026 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5900 IOs, 2340028416 bytes 2023-12-10T19:27:23.176 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6000 IOs, 2386442240 bytes 2023-12-10T19:27:23.281 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6100 IOs, 2404822016 bytes 2023-12-10T19:27:23.405 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6200 IOs, 2437928960 bytes 2023-12-10T19:27:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6300 IOs, 2480016384 bytes 2023-12-10T19:27:23.623 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6400 IOs, 2498316288 bytes 2023-12-10T19:27:23.733 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6500 IOs, 2518684672 bytes 2023-12-10T19:27:23.857 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6600 IOs, 2555770880 bytes 2023-12-10T19:27:23.954 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6700 IOs, 2586536960 bytes 2023-12-10T19:27:24.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6800 IOs, 2619702272 bytes 2023-12-10T19:27:24.215 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:6900 IOs, 2666301440 bytes 2023-12-10T19:27:24.374 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:7000 IOs, 2753356800 bytes 2023-12-10T19:27:24.497 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:7100 IOs, 2768252928 bytes 2023-12-10T19:27:24.633 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:7200 IOs, 2824258560 bytes 2023-12-10T19:27:24.789 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:7300 IOs, 2874308608 bytes 2023-12-10T19:27:24.936 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:7400 IOs, 2923637760 bytes 2023-12-10T19:27:25.159 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:7500 IOs, 3017538560 bytes 2023-12-10T19:27:25.257 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:7600 IOs, 3025329152 bytes 2023-12-10T19:27:25.357 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:7700 IOs, 3063270400 bytes 2023-12-10T19:27:25.513 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:7800 IOs, 3109068800 bytes 2023-12-10T19:27:25.640 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:7900 IOs, 3149810688 bytes 2023-12-10T19:27:25.750 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:8000 IOs, 3172635648 bytes 2023-12-10T19:27:25.916 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:8100 IOs, 3245698048 bytes 2023-12-10T19:27:26.000 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:8200 IOs, 3248504832 bytes 2023-12-10T19:27:26.099 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:8300 IOs, 3281619968 bytes 2023-12-10T19:27:26.145 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=124 2023-12-10T19:27:26.146 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T19:27:26.146 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 124 -eq 124 ] 2023-12-10T19:27:26.146 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:27:26.146 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_present cluster1 mirror image_8 present 2023-12-10T19:27:26.146 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:27:26.146 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:27:26.146 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_8 2023-12-10T19:27:26.146 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=present 2023-12-10T19:27:26.146 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:27:26.146 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:27:26.146 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:27:26.147 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_8 2023-12-10T19:27:26.147 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:27:26.147 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:27:26.147 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_8 2023-12-10T19:27:26.147 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_8 2023-12-10T19:27:26.147 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:27:26.201 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image_id=128b547d6760 2023-12-10T19:27:26.201 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.1 2023-12-10T19:27:26.302 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_8 present 128b547d6760 2023-12-10T19:27:26.302 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:27:26.302 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:27:26.303 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_8 2023-12-10T19:27:26.303 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=present 2023-12-10T19:27:26.303 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id=128b547d6760 2023-12-10T19:27:26.303 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:27:26.303 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:27:26.303 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_8 2023-12-10T19:27:26.303 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:27:26.303 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:27:26.303 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_8 2023-12-10T19:27:26.303 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_8 2023-12-10T19:27:26.303 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:27:26.364 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id=128b547d6760 2023-12-10T19:27:26.364 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 128b547d6760 2023-12-10T19:27:26.364 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -z 128b547d6760 -o 128b547d6760 = 128b547d6760 2023-12-10T19:27:26.364 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_state=present 2023-12-10T19:27:26.364 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test present = present 2023-12-10T19:27:26.364 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:27:26.364 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ disable_mirror cluster2 mirror image_8 2023-12-10T19:27:26.364 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:27:26.364 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:27:26.364 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_8 2023-12-10T19:27:26.364 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_8 2023-12-10T19:27:26.368 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~151s 2023-12-10T19:27:26.857 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi076.stderr:2023-12-10T19:27:26.856+0000 7f3eac23a640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55b33b8da400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=382 2023-12-10T19:27:26.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi076.stderr:2023-12-10T19:27:26.856+0000 7f3eac23a640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55b33b8da400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=386, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=f4a845a0-65d8-4baa-abd4-fd2893f9af83, primary_snap_id=17e, last_copied_object_number=128, snap_seqs={382=18446744073709551614}] 2023-12-10T19:27:31.771 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~157s 2023-12-10T19:27:31.864 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirroring disabled 2023-12-10T19:27:31.877 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2023-12-10T19:27:31.877 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:27:31.877 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:27:31.877 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_8 2023-12-10T19:27:31.877 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=deleted 2023-12-10T19:27:31.877 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:27:31.877 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:27:31.877 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:27:31.877 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_8 2023-12-10T19:27:31.877 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:27:31.877 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:27:31.878 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_8 2023-12-10T19:27:31.878 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_8 2023-12-10T19:27:31.878 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:27:31.922 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_8: (2) No such file or directory 2023-12-10T19:27:31.926 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image_id= 2023-12-10T19:27:31.926 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.1 2023-12-10T19:27:32.028 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_8 deleted 2023-12-10T19:27:32.028 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:27:32.028 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:27:32.028 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_8 2023-12-10T19:27:32.028 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=deleted 2023-12-10T19:27:32.028 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:27:32.028 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:27:32.029 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:27:32.029 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_8 2023-12-10T19:27:32.029 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:27:32.029 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:27:32.029 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_8 2023-12-10T19:27:32.029 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_8 2023-12-10T19:27:32.029 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:27:32.075 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_8: (2) No such file or directory 2023-12-10T19:27:32.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:27:32.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:27:32.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test deleted = deleted 2023-12-10T19:27:32.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:27:32.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ purge_snapshots cluster2 mirror image_8 2023-12-10T19:27:32.079 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:27:32.080 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:27:32.080 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_8 2023-12-10T19:27:32.080 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 snap purge mirror/image_8 2023-12-10T19:27:32.129 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image_retry cluster2 mirror image_8 2023-12-10T19:27:32.129 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:27:32.129 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:27:32.129 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_8 2023-12-10T19:27:32.129 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0 2023-12-10T19:27:32.130 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_8 2023-12-10T19:27:32.130 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:27:32.130 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:27:32.130 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_8 2023-12-10T19:27:32.130 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2023-12-10T19:27:32.177 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2023-12-10T19:27:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 0% complete...failed. 2023-12-10T19:27:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error: image still has watchers 2023-12-10T19:27:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-10T19:27:32.226 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:27:33.227 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_8 2023-12-10T19:27:33.228 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:27:33.228 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:27:33.228 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_8 2023-12-10T19:27:33.228 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2023-12-10T19:27:33.287 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2023-12-10T19:27:33.335 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 0% complete...failed. 2023-12-10T19:27:33.335 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error: image still has watchers 2023-12-10T19:27:33.336 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-10T19:27:33.339 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 2 2023-12-10T19:27:35.340 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_8 2023-12-10T19:27:35.340 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:27:35.340 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:27:35.340 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_8 2023-12-10T19:27:35.340 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2023-12-10T19:27:35.399 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2023-12-10T19:27:35.447 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 0% complete...failed. 2023-12-10T19:27:35.449 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error: image still has watchers 2023-12-10T19:27:35.449 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-10T19:27:35.451 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 4 2023-12-10T19:27:37.175 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~162s 2023-12-10T19:27:39.452 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_8 2023-12-10T19:27:39.452 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:27:39.452 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:27:39.452 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_8 2023-12-10T19:27:39.453 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2023-12-10T19:27:39.511 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2023-12-10T19:27:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 0% complete...failed. 2023-12-10T19:27:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error: image still has watchers 2023-12-10T19:27:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-10T19:27:39.564 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 8 2023-12-10T19:27:42.579 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~168s 2023-12-10T19:27:46.906 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.963514764740564) > weight (0.25) 2023-12-10T19:27:46.907 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.9650878777563792) > weight (0.25) 2023-12-10T19:27:46.907 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.8226530544761558) > weight (0.25) 2023-12-10T19:27:46.907 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.44099258673749075) > weight (0.25) 2023-12-10T19:27:46.907 INFO:tasks.rbd_mirror_thrash:waiting for 64 secs before thrashing 2023-12-10T19:27:47.565 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_8 2023-12-10T19:27:47.565 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:27:47.566 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:27:47.566 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_8 2023-12-10T19:27:47.566 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2023-12-10T19:27:47.623 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2023-12-10T19:27:47.671 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 0% complete...failed. 2023-12-10T19:27:47.672 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error: image still has watchers 2023-12-10T19:27:47.672 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-10T19:27:47.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 16 2023-12-10T19:27:47.983 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~173s 2023-12-10T19:27:53.387 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~178s 2023-12-10T19:27:58.790 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~184s 2023-12-10T19:28:03.677 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ remove_image cluster2 mirror image_8 2023-12-10T19:28:03.678 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:28:03.678 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:28:03.678 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_8 2023-12-10T19:28:03.678 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2023-12-10T19:28:03.732 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2023-12-10T19:28:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-10T19:28:03.961 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:28:03.961 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image=image_9 2023-12-10T19:28:03.961 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ create_image cluster2 mirror image_9 512M 2023-12-10T19:28:03.961 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:28:03.961 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:28:03.961 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:28:03.961 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:28:03.961 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_9 2023-12-10T19:28:03.961 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:28:03.962 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local size=128 2023-12-10T19:28:03.962 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -n 512M ] 2023-12-10T19:28:03.962 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ size=512M 2023-12-10T19:28:03.962 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ shift 2023-12-10T19:28:03.962 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/image_9 2023-12-10T19:28:04.010 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ enable_mirror cluster2 mirror image_9 2023-12-10T19:28:04.010 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:28:04.010 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:28:04.010 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_9 2023-12-10T19:28:04.010 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local mode=snapshot 2023-12-10T19:28:04.010 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_9 snapshot 2023-12-10T19:28:04.193 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~189s 2023-12-10T19:28:05.086 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirroring enabled 2023-12-10T19:28:05.098 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster2 info mirror/image_9 2023-12-10T19:28:05.145 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:rbd image 'image_9': 2023-12-10T19:28:05.145 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: size 512 MiB in 128 objects 2023-12-10T19:28:05.145 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: order 22 (4 MiB objects) 2023-12-10T19:28:05.145 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: snapshot_count: 1 2023-12-10T19:28:05.145 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: 1ac592ca964d 2023-12-10T19:28:05.145 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: block_name_prefix: rbd_data.1ac592ca964d 2023-12-10T19:28:05.145 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: format: 2 2023-12-10T19:28:05.145 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: features: layering 2023-12-10T19:28:05.145 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: op_features: 2023-12-10T19:28:05.145 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: flags: 2023-12-10T19:28:05.145 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: create_timestamp: Sun Dec 10 19:28:04 2023 2023-12-10T19:28:05.145 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: access_timestamp: Sun Dec 10 19:28:04 2023 2023-12-10T19:28:05.145 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: modify_timestamp: Sun Dec 10 19:28:04 2023 2023-12-10T19:28:05.146 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring state: enabled 2023-12-10T19:28:05.146 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring mode: snapshot 2023-12-10T19:28:05.146 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring global id: f22ea0ff-01dc-4ad2-9137-754811a1953d 2023-12-10T19:28:05.146 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mirroring primary: true 2023-12-10T19:28:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stress_write_image cluster2 mirror image_9 2023-12-10T19:28:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T19:28:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:28:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_9 2023-12-10T19:28:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-10T19:28:05.151 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local duration=14 2023-12-10T19:28:05.151 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T19:28:05.151 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2023-12-10T19:28:05.342 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:100 IOs, 60448768 bytes 2023-12-10T19:28:05.442 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:200 IOs, 92317696 bytes 2023-12-10T19:28:05.561 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:300 IOs, 92974080 bytes 2023-12-10T19:28:05.661 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:400 IOs, 102754304 bytes 2023-12-10T19:28:05.803 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:500 IOs, 144615424 bytes 2023-12-10T19:28:05.963 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:600 IOs, 203943936 bytes 2023-12-10T19:28:06.075 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:700 IOs, 233582592 bytes 2023-12-10T19:28:06.168 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:800 IOs, 239406080 bytes 2023-12-10T19:28:06.286 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:900 IOs, 272675840 bytes 2023-12-10T19:28:06.409 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1000 IOs, 305772544 bytes 2023-12-10T19:28:06.518 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1100 IOs, 327419904 bytes 2023-12-10T19:28:06.659 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1200 IOs, 391879680 bytes 2023-12-10T19:28:06.885 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1300 IOs, 494698496 bytes 2023-12-10T19:28:06.903 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.43307293725882867) > weight (0.25) 2023-12-10T19:28:06.903 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2023-12-10T19:28:06.903 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2023-12-10T19:28:06.903 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2023-12-10T19:28:06.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi076.stderr:2023-12-10T19:28:06.907+0000 7f3eb9a55640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 49556) UID: 1000 2023-12-10T19:28:06.968 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1400 IOs, 495312896 bytes 2023-12-10T19:28:07.101 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1500 IOs, 534223872 bytes 2023-12-10T19:28:07.245 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1600 IOs, 570881024 bytes 2023-12-10T19:28:07.348 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1700 IOs, 601196544 bytes 2023-12-10T19:28:07.460 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1800 IOs, 601810944 bytes 2023-12-10T19:28:07.553 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:1900 IOs, 619757568 bytes 2023-12-10T19:28:07.649 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2000 IOs, 620398592 bytes 2023-12-10T19:28:07.824 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2100 IOs, 690971648 bytes 2023-12-10T19:28:07.992 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2200 IOs, 733753344 bytes 2023-12-10T19:28:08.137 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2300 IOs, 793183232 bytes 2023-12-10T19:28:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2400 IOs, 863257600 bytes 2023-12-10T19:28:08.529 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2500 IOs, 934104064 bytes 2023-12-10T19:28:08.634 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2600 IOs, 957806592 bytes 2023-12-10T19:28:08.796 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2700 IOs, 999963648 bytes 2023-12-10T19:28:08.910 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2800 IOs, 1020546048 bytes 2023-12-10T19:28:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:2900 IOs, 1092110336 bytes 2023-12-10T19:28:09.236 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3000 IOs, 1133027328 bytes 2023-12-10T19:28:09.353 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3100 IOs, 1151882240 bytes 2023-12-10T19:28:09.475 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3200 IOs, 1173339136 bytes 2023-12-10T19:28:09.597 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~195s 2023-12-10T19:28:09.630 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3300 IOs, 1226191872 bytes 2023-12-10T19:28:09.775 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3400 IOs, 1288091648 bytes 2023-12-10T19:28:09.897 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3500 IOs, 1318531072 bytes 2023-12-10T19:28:10.039 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3600 IOs, 1382080512 bytes 2023-12-10T19:28:10.185 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3700 IOs, 1401245696 bytes 2023-12-10T19:28:10.272 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3800 IOs, 1403499520 bytes 2023-12-10T19:28:10.464 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:3900 IOs, 1477110784 bytes 2023-12-10T19:28:10.563 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4000 IOs, 1492215808 bytes 2023-12-10T19:28:10.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4100 IOs, 1568942080 bytes 2023-12-10T19:28:10.888 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4200 IOs, 1617809408 bytes 2023-12-10T19:28:11.020 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4300 IOs, 1645553664 bytes 2023-12-10T19:28:11.154 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4400 IOs, 1684259840 bytes 2023-12-10T19:28:11.241 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4500 IOs, 1687971840 bytes 2023-12-10T19:28:11.344 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4600 IOs, 1711475712 bytes 2023-12-10T19:28:11.461 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4700 IOs, 1733745664 bytes 2023-12-10T19:28:11.607 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4800 IOs, 1784927232 bytes 2023-12-10T19:28:11.714 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:4900 IOs, 1804406784 bytes 2023-12-10T19:28:11.838 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5000 IOs, 1827708928 bytes 2023-12-10T19:28:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5100 IOs, 1832649728 bytes 2023-12-10T19:28:12.028 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5200 IOs, 1838112768 bytes 2023-12-10T19:28:12.192 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5300 IOs, 1897656320 bytes 2023-12-10T19:28:12.330 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5400 IOs, 1942386688 bytes 2023-12-10T19:28:12.480 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5500 IOs, 1985092608 bytes 2023-12-10T19:28:12.627 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5600 IOs, 2030379008 bytes 2023-12-10T19:28:12.751 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:5700 IOs, 2062826496 bytes 2023-12-10T19:28:12.807 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr:2023-12-10T19:28:12.811+0000 7fa376a01640 -1 bdev(0x555750fe2e00 /var/lib/ceph/osd/cluster2-1/block) _aio_thread got r=-28 ((28) No space left on device) 2023-12-10T19:28:12.809 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr:./src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7fa376a01640 time 2023-12-10T19:28:12.813450+0000 2023-12-10T19:28:12.810 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr:./src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-12-10T19:28:12.810 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: ceph version 19.0.0-130-g3d2db44f (3d2db44fa991bc355074aa9b7647f63b77201fef) reef (dev) 2023-12-10T19:28:12.810 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x55574ee4f750] 2023-12-10T19:28:12.810 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: 2: (KernelDevice::_aio_thread()+0x9bc) [0x55574f7377fc] 2023-12-10T19:28:12.810 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: 3: ceph-osd(+0x15b40e1) [0x55574f71a0e1] 2023-12-10T19:28:12.810 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: 4: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fa38484bb43] 2023-12-10T19:28:12.810 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: 5: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fa3848dda00] 2023-12-10T19:28:12.810 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr:2023-12-10T19:28:12.815+0000 7fa376a01640 -1 ./src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7fa376a01640 time 2023-12-10T19:28:12.813450+0000 2023-12-10T19:28:12.810 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr:./src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-12-10T19:28:12.810 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: 2023-12-10T19:28:12.810 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: ceph version 19.0.0-130-g3d2db44f (3d2db44fa991bc355074aa9b7647f63b77201fef) reef (dev) 2023-12-10T19:28:12.810 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x55574ee4f750] 2023-12-10T19:28:12.810 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: 2: (KernelDevice::_aio_thread()+0x9bc) [0x55574f7377fc] 2023-12-10T19:28:12.810 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: 3: ceph-osd(+0x15b40e1) [0x55574f71a0e1] 2023-12-10T19:28:12.811 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: 4: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fa38484bb43] 2023-12-10T19:28:12.811 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: 5: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fa3848dda00] 2023-12-10T19:28:12.811 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: 2023-12-10T19:28:12.811 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr:*** Caught signal (Aborted) ** 2023-12-10T19:28:12.811 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: in thread 7fa376a01640 thread_name:bstore_aio 2023-12-10T19:28:12.812 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: ceph version 19.0.0-130-g3d2db44f (3d2db44fa991bc355074aa9b7647f63b77201fef) reef (dev) 2023-12-10T19:28:12.812 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fa3847f9520] 2023-12-10T19:28:12.812 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: 2: pthread_kill() 2023-12-10T19:28:12.812 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: 3: raise() 2023-12-10T19:28:12.812 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: 4: abort() 2023-12-10T19:28:12.812 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x17b) [0x55574ee4f805] 2023-12-10T19:28:12.812 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: 6: (KernelDevice::_aio_thread()+0x9bc) [0x55574f7377fc] 2023-12-10T19:28:12.812 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: 7: ceph-osd(+0x15b40e1) [0x55574f71a0e1] 2023-12-10T19:28:12.812 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: 8: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fa38484bb43] 2023-12-10T19:28:12.813 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: 9: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fa3848dda00] 2023-12-10T19:28:12.813 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr:2023-12-10T19:28:12.815+0000 7fa376a01640 -1 *** Caught signal (Aborted) ** 2023-12-10T19:28:12.813 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: in thread 7fa376a01640 thread_name:bstore_aio 2023-12-10T19:28:12.813 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: 2023-12-10T19:28:12.813 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: ceph version 19.0.0-130-g3d2db44f (3d2db44fa991bc355074aa9b7647f63b77201fef) reef (dev) 2023-12-10T19:28:12.813 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fa3847f9520] 2023-12-10T19:28:12.813 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: 2: pthread_kill() 2023-12-10T19:28:12.813 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: 3: raise() 2023-12-10T19:28:12.813 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: 4: abort() 2023-12-10T19:28:12.813 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x17b) [0x55574ee4f805] 2023-12-10T19:28:12.813 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: 6: (KernelDevice::_aio_thread()+0x9bc) [0x55574f7377fc] 2023-12-10T19:28:12.813 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: 7: ceph-osd(+0x15b40e1) [0x55574f71a0e1] 2023-12-10T19:28:12.814 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: 8: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fa38484bb43] 2023-12-10T19:28:12.814 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: 9: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fa3848dda00] 2023-12-10T19:28:12.814 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-10T19:28:12.814 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: 2023-12-10T19:28:12.824 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: -1756> 2023-12-10T19:28:12.811+0000 7fa376a01640 -1 bdev(0x555750fe2e00 /var/lib/ceph/osd/cluster2-1/block) _aio_thread got r=-28 ((28) No space left on device) 2023-12-10T19:28:12.825 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: -1257> 2023-12-10T19:28:12.815+0000 7fa376a01640 -1 ./src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7fa376a01640 time 2023-12-10T19:28:12.813450+0000 2023-12-10T19:28:12.825 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr:./src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-12-10T19:28:12.825 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: 2023-12-10T19:28:12.825 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: ceph version 19.0.0-130-g3d2db44f (3d2db44fa991bc355074aa9b7647f63b77201fef) reef (dev) 2023-12-10T19:28:12.825 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x55574ee4f750] 2023-12-10T19:28:12.825 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: 2: (KernelDevice::_aio_thread()+0x9bc) [0x55574f7377fc] 2023-12-10T19:28:12.825 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: 3: ceph-osd(+0x15b40e1) [0x55574f71a0e1] 2023-12-10T19:28:12.825 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: 4: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fa38484bb43] 2023-12-10T19:28:12.825 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: 5: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fa3848dda00] 2023-12-10T19:28:12.825 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: 2023-12-10T19:28:12.826 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: -5> 2023-12-10T19:28:12.815+0000 7fa376a01640 -1 *** Caught signal (Aborted) ** 2023-12-10T19:28:12.826 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: in thread 7fa376a01640 thread_name:bstore_aio 2023-12-10T19:28:12.826 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: 2023-12-10T19:28:12.826 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: ceph version 19.0.0-130-g3d2db44f (3d2db44fa991bc355074aa9b7647f63b77201fef) reef (dev) 2023-12-10T19:28:12.826 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fa3847f9520] 2023-12-10T19:28:12.826 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: 2: pthread_kill() 2023-12-10T19:28:12.826 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: 3: raise() 2023-12-10T19:28:12.826 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: 4: abort() 2023-12-10T19:28:12.826 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x17b) [0x55574ee4f805] 2023-12-10T19:28:12.826 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: 6: (KernelDevice::_aio_thread()+0x9bc) [0x55574f7377fc] 2023-12-10T19:28:12.826 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: 7: ceph-osd(+0x15b40e1) [0x55574f71a0e1] 2023-12-10T19:28:12.826 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: 8: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fa38484bb43] 2023-12-10T19:28:12.826 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: 9: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fa3848dda00] 2023-12-10T19:28:12.827 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-10T19:28:12.827 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: 2023-12-10T19:28:12.838 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: -1868> 2023-12-10T19:28:12.811+0000 7fa376a01640 -1 bdev(0x555750fe2e00 /var/lib/ceph/osd/cluster2-1/block) _aio_thread got r=-28 ((28) No space left on device) 2023-12-10T19:28:12.838 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: -1369> 2023-12-10T19:28:12.815+0000 7fa376a01640 -1 ./src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7fa376a01640 time 2023-12-10T19:28:12.813450+0000 2023-12-10T19:28:12.838 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr:./src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-12-10T19:28:12.838 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: 2023-12-10T19:28:12.838 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: ceph version 19.0.0-130-g3d2db44f (3d2db44fa991bc355074aa9b7647f63b77201fef) reef (dev) 2023-12-10T19:28:12.838 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x55574ee4f750] 2023-12-10T19:28:12.838 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: 2: (KernelDevice::_aio_thread()+0x9bc) [0x55574f7377fc] 2023-12-10T19:28:12.838 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: 3: ceph-osd(+0x15b40e1) [0x55574f71a0e1] 2023-12-10T19:28:12.838 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: 4: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fa38484bb43] 2023-12-10T19:28:12.838 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: 5: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fa3848dda00] 2023-12-10T19:28:12.838 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: 2023-12-10T19:28:12.838 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: -117> 2023-12-10T19:28:12.815+0000 7fa376a01640 -1 *** Caught signal (Aborted) ** 2023-12-10T19:28:12.838 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: in thread 7fa376a01640 thread_name:bstore_aio 2023-12-10T19:28:12.838 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: 2023-12-10T19:28:12.838 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: ceph version 19.0.0-130-g3d2db44f (3d2db44fa991bc355074aa9b7647f63b77201fef) reef (dev) 2023-12-10T19:28:12.839 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fa3847f9520] 2023-12-10T19:28:12.839 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: 2: pthread_kill() 2023-12-10T19:28:12.839 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: 3: raise() 2023-12-10T19:28:12.839 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: 4: abort() 2023-12-10T19:28:12.839 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x17b) [0x55574ee4f805] 2023-12-10T19:28:12.839 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: 6: (KernelDevice::_aio_thread()+0x9bc) [0x55574f7377fc] 2023-12-10T19:28:12.839 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: 7: ceph-osd(+0x15b40e1) [0x55574f71a0e1] 2023-12-10T19:28:12.839 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: 8: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fa38484bb43] 2023-12-10T19:28:12.839 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: 9: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fa3848dda00] 2023-12-10T19:28:12.839 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-10T19:28:12.839 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr: 2023-12-10T19:28:12.904 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2023-12-10T19:28:12.904 INFO:teuthology.orchestra.run:waiting for 600 2023-12-10T19:28:12.904 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2023-12-10T19:28:12.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-12-10T19:28:12.905 DEBUG:teuthology.orchestra.run.smithi076:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-12-10T19:28:12.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-12-10T19:28:12.907 INFO:tasks.rbd_mirror_thrash:waiting for 117 secs before thrashing 2023-12-10T19:28:13.169 INFO:tasks.ceph.cluster2.osd.1.smithi076.stderr:daemon-helper: command failed with exit status 5 2023-12-10T19:28:14.900 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~200s 2023-12-10T19:28:14.901 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~0s 2023-12-10T19:28:19.157 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ error_code=124 2023-12-10T19:28:19.157 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set -e 2023-12-10T19:28:19.157 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 124 -eq 124 ] 2023-12-10T19:28:19.157 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 0 2023-12-10T19:28:19.157 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ wait_for_image_present cluster1 mirror image_9 present 2023-12-10T19:28:19.157 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:28:19.157 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:28:19.157 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_9 2023-12-10T19:28:19.157 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local state=present 2023-12-10T19:28:19.157 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:28:19.158 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local s 2023-12-10T19:28:19.158 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:28:19.158 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_9 2023-12-10T19:28:19.158 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:28:19.158 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:28:19.158 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_9 2023-12-10T19:28:19.158 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_9 2023-12-10T19:28:19.158 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:28:19.200 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_9: (2) No such file or directory 2023-12-10T19:28:19.204 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ image_id= 2023-12-10T19:28:19.205 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 0.1 2023-12-10T19:28:19.306 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_9 present 2023-12-10T19:28:19.306 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:28:19.306 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:28:19.306 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_9 2023-12-10T19:28:19.306 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=present 2023-12-10T19:28:19.306 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:28:19.307 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:28:19.307 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:28:19.307 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_9 2023-12-10T19:28:19.307 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:28:19.307 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:28:19.307 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_9 2023-12-10T19:28:19.307 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ + rbd --cluster=cluster1 info mirror/image_9 2023-12-10T19:28:19.307 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:28:19.354 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_9: (2) No such file or directory 2023-12-10T19:28:19.358 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:28:19.358 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:28:19.358 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test present = deleted 2023-12-10T19:28:19.358 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 1 2023-12-10T19:28:20.204 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~205s 2023-12-10T19:28:20.204 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~5s 2023-12-10T19:28:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_9 present 2023-12-10T19:28:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:28:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:28:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_9 2023-12-10T19:28:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=present 2023-12-10T19:28:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:28:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:28:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:28:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_9 2023-12-10T19:28:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:28:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:28:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_9 2023-12-10T19:28:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_9 2023-12-10T19:28:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:28:20.406 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_9: (2) No such file or directory 2023-12-10T19:28:20.410 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:28:20.410 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:28:20.410 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test present = deleted 2023-12-10T19:28:20.410 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 2 2023-12-10T19:28:22.412 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_9 present 2023-12-10T19:28:22.412 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:28:22.412 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:28:22.412 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_9 2023-12-10T19:28:22.412 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=present 2023-12-10T19:28:22.412 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:28:22.412 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:28:22.412 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:28:22.412 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_9 2023-12-10T19:28:22.413 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:28:22.413 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:28:22.413 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_9 2023-12-10T19:28:22.413 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_9 2023-12-10T19:28:22.413 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:28:22.461 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_9: (2) No such file or directory 2023-12-10T19:28:22.463 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:28:22.463 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:28:22.463 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test present = deleted 2023-12-10T19:28:22.463 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 4 2023-12-10T19:28:25.507 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~211s 2023-12-10T19:28:25.507 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~11s 2023-12-10T19:28:26.465 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_9 present 2023-12-10T19:28:26.465 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:28:26.465 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:28:26.465 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_9 2023-12-10T19:28:26.465 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=present 2023-12-10T19:28:26.465 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:28:26.466 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:28:26.466 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:28:26.466 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_9 2023-12-10T19:28:26.466 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:28:26.466 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:28:26.466 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_9 2023-12-10T19:28:26.466 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_9 2023-12-10T19:28:26.466 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:28:26.511 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_9: (2) No such file or directory 2023-12-10T19:28:26.515 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:28:26.515 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:28:26.515 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test present = deleted 2023-12-10T19:28:26.515 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 8 2023-12-10T19:28:30.810 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~216s 2023-12-10T19:28:30.810 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~16s 2023-12-10T19:28:34.517 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_9 present 2023-12-10T19:28:34.517 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:28:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:28:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_9 2023-12-10T19:28:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=present 2023-12-10T19:28:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:28:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:28:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:28:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_9 2023-12-10T19:28:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:28:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:28:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_9 2023-12-10T19:28:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_9 2023-12-10T19:28:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:28:34.566 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_9: (2) No such file or directory 2023-12-10T19:28:34.570 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:28:34.570 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:28:34.570 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test present = deleted 2023-12-10T19:28:34.570 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 8 2023-12-10T19:28:36.112 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~221s 2023-12-10T19:28:36.113 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~21s 2023-12-10T19:28:41.415 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~226s 2023-12-10T19:28:41.416 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~27s 2023-12-10T19:28:42.573 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_9 present 2023-12-10T19:28:42.573 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:28:42.573 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:28:42.573 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_9 2023-12-10T19:28:42.573 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=present 2023-12-10T19:28:42.574 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:28:42.574 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:28:42.574 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:28:42.574 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_9 2023-12-10T19:28:42.574 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:28:42.574 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:28:42.574 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_9 2023-12-10T19:28:42.574 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_9 2023-12-10T19:28:42.575 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:28:42.618 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_9: (2) No such file or directory 2023-12-10T19:28:42.622 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:28:42.622 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:28:42.622 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test present = deleted 2023-12-10T19:28:42.622 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 8 2023-12-10T19:28:46.718 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~232s 2023-12-10T19:28:46.719 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~32s 2023-12-10T19:28:50.624 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_9 present 2023-12-10T19:28:50.624 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:28:50.624 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:28:50.624 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_9 2023-12-10T19:28:50.624 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=present 2023-12-10T19:28:50.624 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:28:50.624 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:28:50.624 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:28:50.625 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_9 2023-12-10T19:28:50.625 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:28:50.625 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:28:50.625 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_9 2023-12-10T19:28:50.625 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_9 2023-12-10T19:28:50.625 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:28:50.669 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_9: (2) No such file or directory 2023-12-10T19:28:50.673 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:28:50.673 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:28:50.673 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test present = deleted 2023-12-10T19:28:50.673 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 8 2023-12-10T19:28:50.908 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.4451675750049149) > weight (0.25) 2023-12-10T19:28:50.908 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2023-12-10T19:28:50.908 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2023-12-10T19:28:50.908 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2023-12-10T19:28:50.910 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr:2023-12-10T19:28:50.914+0000 7f2c61de2640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 62746) UID: 1000 2023-12-10T19:28:52.022 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~237s 2023-12-10T19:28:52.022 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~37s 2023-12-10T19:28:52.909 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2023-12-10T19:28:52.909 INFO:teuthology.orchestra.run:waiting for 600 2023-12-10T19:28:57.325 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~242s 2023-12-10T19:28:57.325 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~42s 2023-12-10T19:28:58.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_9 present 2023-12-10T19:28:58.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:28:58.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:28:58.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_9 2023-12-10T19:28:58.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=present 2023-12-10T19:28:58.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:28:58.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:28:58.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:28:58.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_9 2023-12-10T19:28:58.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:28:58.676 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:28:58.676 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_9 2023-12-10T19:28:58.676 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_9 2023-12-10T19:28:58.676 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:28:58.723 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_9: (2) No such file or directory 2023-12-10T19:28:58.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:28:58.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:28:58.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test present = deleted 2023-12-10T19:28:58.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 8 2023-12-10T19:29:02.630 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~248s 2023-12-10T19:29:02.630 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~48s 2023-12-10T19:29:06.729 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_9 present 2023-12-10T19:29:06.729 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:29:06.729 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:29:06.729 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_9 2023-12-10T19:29:06.729 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=present 2023-12-10T19:29:06.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:29:06.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:29:06.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:29:06.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_9 2023-12-10T19:29:06.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:29:06.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:29:06.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_9 2023-12-10T19:29:06.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_9 2023-12-10T19:29:06.730 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:29:06.776 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_9: (2) No such file or directory 2023-12-10T19:29:06.780 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:29:06.780 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:29:06.780 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test present = deleted 2023-12-10T19:29:06.780 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 8 2023-12-10T19:29:07.933 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~253s 2023-12-10T19:29:07.933 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~53s 2023-12-10T19:29:13.237 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~258s 2023-12-10T19:29:13.237 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~58s 2023-12-10T19:29:14.782 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_9 present 2023-12-10T19:29:14.782 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:29:14.782 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:29:14.782 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_9 2023-12-10T19:29:14.782 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=present 2023-12-10T19:29:14.782 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:29:14.782 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:29:14.782 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:29:14.783 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_9 2023-12-10T19:29:14.783 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:29:14.783 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:29:14.783 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_9 2023-12-10T19:29:14.783 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_9 2023-12-10T19:29:14.783 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:29:14.830 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_9: (2) No such file or directory 2023-12-10T19:29:14.833 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:29:14.833 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:29:14.833 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test present = deleted 2023-12-10T19:29:14.834 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 8 2023-12-10T19:29:18.540 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~264s 2023-12-10T19:29:18.540 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~64s 2023-12-10T19:29:22.836 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_9 present 2023-12-10T19:29:22.837 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:29:22.837 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:29:22.837 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_9 2023-12-10T19:29:22.837 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=present 2023-12-10T19:29:22.837 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:29:22.837 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:29:22.837 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:29:22.837 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_9 2023-12-10T19:29:22.837 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:29:22.838 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:29:22.838 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_9 2023-12-10T19:29:22.838 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_9 2023-12-10T19:29:22.838 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:29:22.882 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_9: (2) No such file or directory 2023-12-10T19:29:22.888 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:29:22.888 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:29:22.888 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test present = deleted 2023-12-10T19:29:22.889 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 8 2023-12-10T19:29:23.846 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~269s 2023-12-10T19:29:23.846 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~69s 2023-12-10T19:29:29.149 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~274s 2023-12-10T19:29:29.149 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~74s 2023-12-10T19:29:30.889 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_9 present 2023-12-10T19:29:30.889 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:29:30.889 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:29:30.889 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_9 2023-12-10T19:29:30.889 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=present 2023-12-10T19:29:30.889 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:29:30.889 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:29:30.889 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:29:30.890 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_9 2023-12-10T19:29:30.890 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:29:30.890 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:29:30.890 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_9 2023-12-10T19:29:30.890 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_9 2023-12-10T19:29:30.890 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:29:30.937 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_9: (2) No such file or directory 2023-12-10T19:29:30.940 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:29:30.940 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:29:30.941 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test present = deleted 2023-12-10T19:29:30.941 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 16 2023-12-10T19:29:34.452 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~279s 2023-12-10T19:29:34.453 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~80s 2023-12-10T19:29:39.755 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~285s 2023-12-10T19:29:39.755 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~85s 2023-12-10T19:29:45.059 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~290s 2023-12-10T19:29:45.059 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~90s 2023-12-10T19:29:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_9 present 2023-12-10T19:29:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:29:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:29:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_9 2023-12-10T19:29:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=present 2023-12-10T19:29:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:29:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:29:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:29:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_9 2023-12-10T19:29:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:29:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:29:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_9 2023-12-10T19:29:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_9 2023-12-10T19:29:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:29:46.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_9: (2) No such file or directory 2023-12-10T19:29:46.994 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:29:46.994 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:29:46.994 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test present = deleted 2023-12-10T19:29:46.994 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 16 2023-12-10T19:29:50.362 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~295s 2023-12-10T19:29:50.362 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~95s 2023-12-10T19:29:55.666 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~301s 2023-12-10T19:29:55.666 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~101s 2023-12-10T19:29:55.666 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2023-12-10T19:29:55.969 INFO:tasks.ceph.cluster2.osd.2:Sent signal 15 2023-12-10T19:29:55.969 INFO:tasks.ceph.cluster2.mon.a:Sent signal 15 2023-12-10T19:29:55.970 INFO:tasks.ceph.cluster2.mgr.x:Sent signal 15 2023-12-10T19:29:55.970 INFO:tasks.ceph.cluster2.osd.2.smithi076.stderr:2023-12-10T19:29:55.975+0000 7f7f78b74640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 (PID: 34993) UID: 0 2023-12-10T19:29:55.970 INFO:tasks.ceph.cluster2.osd.2.smithi076.stderr:2023-12-10T19:29:55.975+0000 7f7f78b74640 -1 osd.2 379 *** Got signal Terminated *** 2023-12-10T19:29:55.970 INFO:tasks.ceph.cluster2.osd.2.smithi076.stderr:2023-12-10T19:29:55.975+0000 7f7f78b74640 -1 osd.2 379 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-10T19:29:55.971 INFO:tasks.ceph.cluster2.mon.a.smithi076.stderr:2023-12-10T19:29:55.975+0000 7fdaf06d5640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mon -f --cluster cluster2 -i a (PID: 34842) UID: 0 2023-12-10T19:29:55.971 INFO:tasks.ceph.cluster2.mon.a.smithi076.stderr:2023-12-10T19:29:55.975+0000 7fdaf06d5640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-12-10T19:29:55.971 INFO:tasks.ceph.cluster2.mgr.x.smithi033.stderr:2023-12-10T19:29:55.968+0000 7ff867597640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mgr -f --cluster cluster2 -i x (PID: 39452) UID: 0 2023-12-10T19:29:55.972 INFO:tasks.ceph.cluster2.mgr.x.smithi033.stderr:2023-12-10T19:29:55.968+0000 7ff867597640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2023-12-10T19:30:02.996 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_9 present 2023-12-10T19:30:02.996 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:30:02.996 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:30:02.996 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_9 2023-12-10T19:30:02.996 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=present 2023-12-10T19:30:02.996 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:30:02.997 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:30:02.997 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:30:02.997 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_9 2023-12-10T19:30:02.997 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:30:02.997 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:30:02.997 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_9 2023-12-10T19:30:02.997 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_9 2023-12-10T19:30:02.997 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:30:03.043 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_9: (2) No such file or directory 2023-12-10T19:30:03.047 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:30:03.047 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:30:03.047 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test present = deleted 2023-12-10T19:30:03.047 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 32 2023-12-10T19:30:09.908 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6751733960229261) > weight (0.25) 2023-12-10T19:30:09.908 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.9519201899650795) > weight (0.25) 2023-12-10T19:30:09.909 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.702265551372651) > weight (0.25) 2023-12-10T19:30:09.909 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.8599363094947838) > weight (0.25) 2023-12-10T19:30:09.909 INFO:tasks.rbd_mirror_thrash:waiting for 73 secs before thrashing 2023-12-10T19:30:35.050 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_9 present 2023-12-10T19:30:35.050 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:30:35.050 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:30:35.051 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_9 2023-12-10T19:30:35.051 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=present 2023-12-10T19:30:35.051 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:30:35.051 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:30:35.051 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:30:35.051 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_9 2023-12-10T19:30:35.051 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:30:35.051 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:30:35.051 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_9 2023-12-10T19:30:35.051 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_9 2023-12-10T19:30:35.051 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:30:35.095 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_9: (2) No such file or directory 2023-12-10T19:30:35.100 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:30:35.100 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:30:35.100 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test present = deleted 2023-12-10T19:30:35.100 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sleep 32 2023-12-10T19:31:07.102 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test_image_present cluster1 mirror image_9 present 2023-12-10T19:31:07.103 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:31:07.103 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:31:07.103 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_9 2023-12-10T19:31:07.103 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local test_state=present 2023-12-10T19:31:07.103 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image_id= 2023-12-10T19:31:07.103 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_state=deleted 2023-12-10T19:31:07.103 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local current_image_id 2023-12-10T19:31:07.103 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ get_image_id cluster1 mirror image_9 2023-12-10T19:31:07.103 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T19:31:07.103 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pool=mirror 2023-12-10T19:31:07.103 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local image=image_9 2023-12-10T19:31:07.103 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster=cluster1 info mirror/image_9 2023-12-10T19:31:07.103 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-10T19:31:07.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: error opening image image_9: (2) No such file or directory 2023-12-10T19:31:07.153 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ current_image_id= 2023-12-10T19:31:07.153 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 2023-12-10T19:31:07.153 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test present = deleted 2023-12-10T19:31:07.153 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 1 2023-12-10T19:31:07.153 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cleanup 1 2023-12-10T19:31:07.153 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local error_code=1 2023-12-10T19:31:07.153 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set +e 2023-12-10T19:31:07.153 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 -ne 0 ] 2023-12-10T19:31:07.153 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ status 2023-12-10T19:31:07.153 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster daemon image_pool image_ns image 2023-12-10T19:31:07.154 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster1 status 2023-12-10T19:31:07.154 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1 status 2023-12-10T19:31:07.154 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ CEPH_ARGS= ceph --cluster cluster1 -s 2023-12-10T19:31:07.624 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: cluster: 2023-12-10T19:31:07.624 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: id: d3c37dcf-9605-4f22-911b-14b903774172 2023-12-10T19:31:07.624 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: health: HEALTH_OK 2023-12-10T19:31:07.624 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:07.624 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: services: 2023-12-10T19:31:07.624 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mon: 1 daemons, quorum a (age 40m) 2023-12-10T19:31:07.624 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: mgr: x(active, since 40m) 2023-12-10T19:31:07.624 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: osd: 3 osds: 3 up (since 40m), 3 in (since 40m) 2023-12-10T19:31:07.625 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: rbd-mirror: 4 daemons active (1 hosts) 2023-12-10T19:31:07.625 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:07.625 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: data: 2023-12-10T19:31:07.625 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: pools: 4 pools, 137 pgs 2023-12-10T19:31:07.625 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: objects: 35 objects, 577 KiB 2023-12-10T19:31:07.625 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: usage: 292 MiB used, 270 GiB / 270 GiB avail 2023-12-10T19:31:07.625 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: pgs: 137 active+clean 2023-12-10T19:31:07.625 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:07.625 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: io: 2023-12-10T19:31:07.625 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: client: 2.3 KiB/s rd, 341 B/s wr, 2 op/s rd, 0 op/s wr 2023-12-10T19:31:07.625 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:07.639 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ CEPH_ARGS= ceph --cluster cluster1 service dump 2023-12-10T19:31:08.015 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:{ 2023-12-10T19:31:08.015 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "epoch": 42, 2023-12-10T19:31:08.015 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "modified": "2023-12-10T19:28:13.696201+0000", 2023-12-10T19:31:08.015 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "services": { 2023-12-10T19:31:08.015 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "rbd-mirror": { 2023-12-10T19:31:08.015 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "daemons": { 2023-12-10T19:31:08.015 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "summary": "", 2023-12-10T19:31:08.015 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "5246": { 2023-12-10T19:31:08.016 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "start_epoch": 30, 2023-12-10T19:31:08.016 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "start_stamp": "2023-12-10T19:15:30.971452+0000", 2023-12-10T19:31:08.016 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "gid": 5246, 2023-12-10T19:31:08.016 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "addr": "172.21.15.76:0/1286799822", 2023-12-10T19:31:08.016 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "metadata": { 2023-12-10T19:31:08.016 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "arch": "x86_64", 2023-12-10T19:31:08.016 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "ceph_release": "reef", 2023-12-10T19:31:08.016 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "ceph_version": "ceph version 19.0.0-130-g3d2db44f (3d2db44fa991bc355074aa9b7647f63b77201fef) reef (dev)", 2023-12-10T19:31:08.016 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "ceph_version_short": "19.0.0-130-g3d2db44f", 2023-12-10T19:31:08.016 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2023-12-10T19:31:08.016 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "distro": "ubuntu", 2023-12-10T19:31:08.016 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2023-12-10T19:31:08.016 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "distro_version": "22.04", 2023-12-10T19:31:08.016 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "hostname": "smithi076", 2023-12-10T19:31:08.016 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "id": "mirror.2", 2023-12-10T19:31:08.017 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "instance_id": "5246", 2023-12-10T19:31:08.017 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "kernel_description": "#101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023", 2023-12-10T19:31:08.017 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "kernel_version": "5.15.0-91-generic", 2023-12-10T19:31:08.017 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "mem_swap_kb": "2097148", 2023-12-10T19:31:08.017 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "mem_total_kb": "32746948", 2023-12-10T19:31:08.017 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "os": "Linux" 2023-12-10T19:31:08.017 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: }, 2023-12-10T19:31:08.017 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "task_status": {} 2023-12-10T19:31:08.017 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: }, 2023-12-10T19:31:08.017 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "6041": { 2023-12-10T19:31:08.017 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "start_epoch": 38, 2023-12-10T19:31:08.017 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "start_stamp": "2023-12-10T19:21:45.996492+0000", 2023-12-10T19:31:08.017 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "gid": 6041, 2023-12-10T19:31:08.017 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "addr": "172.21.15.76:0/2547579397", 2023-12-10T19:31:08.017 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "metadata": { 2023-12-10T19:31:08.018 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "arch": "x86_64", 2023-12-10T19:31:08.018 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "ceph_release": "reef", 2023-12-10T19:31:08.018 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "ceph_version": "ceph version 19.0.0-130-g3d2db44f (3d2db44fa991bc355074aa9b7647f63b77201fef) reef (dev)", 2023-12-10T19:31:08.018 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "ceph_version_short": "19.0.0-130-g3d2db44f", 2023-12-10T19:31:08.018 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2023-12-10T19:31:08.018 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "distro": "ubuntu", 2023-12-10T19:31:08.018 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2023-12-10T19:31:08.018 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "distro_version": "22.04", 2023-12-10T19:31:08.018 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "hostname": "smithi076", 2023-12-10T19:31:08.018 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "id": "mirror.0", 2023-12-10T19:31:08.018 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "instance_id": "6041", 2023-12-10T19:31:08.018 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "kernel_description": "#101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023", 2023-12-10T19:31:08.018 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "kernel_version": "5.15.0-91-generic", 2023-12-10T19:31:08.018 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "mem_swap_kb": "2097148", 2023-12-10T19:31:08.018 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "mem_total_kb": "32746948", 2023-12-10T19:31:08.019 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "os": "Linux" 2023-12-10T19:31:08.019 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: }, 2023-12-10T19:31:08.019 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "task_status": {} 2023-12-10T19:31:08.019 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: }, 2023-12-10T19:31:08.019 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "6082": { 2023-12-10T19:31:08.019 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "start_epoch": 40, 2023-12-10T19:31:08.019 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "start_stamp": "2023-12-10T19:23:22.026126+0000", 2023-12-10T19:31:08.019 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "gid": 6082, 2023-12-10T19:31:08.019 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "addr": "172.21.15.76:0/3272982280", 2023-12-10T19:31:08.019 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "metadata": { 2023-12-10T19:31:08.019 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "arch": "x86_64", 2023-12-10T19:31:08.019 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "ceph_release": "reef", 2023-12-10T19:31:08.019 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "ceph_version": "ceph version 19.0.0-130-g3d2db44f (3d2db44fa991bc355074aa9b7647f63b77201fef) reef (dev)", 2023-12-10T19:31:08.019 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "ceph_version_short": "19.0.0-130-g3d2db44f", 2023-12-10T19:31:08.019 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2023-12-10T19:31:08.020 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "distro": "ubuntu", 2023-12-10T19:31:08.020 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2023-12-10T19:31:08.020 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "distro_version": "22.04", 2023-12-10T19:31:08.020 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "hostname": "smithi076", 2023-12-10T19:31:08.020 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "id": "mirror.3", 2023-12-10T19:31:08.020 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "instance_id": "6082", 2023-12-10T19:31:08.020 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "kernel_description": "#101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023", 2023-12-10T19:31:08.020 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "kernel_version": "5.15.0-91-generic", 2023-12-10T19:31:08.020 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "mem_swap_kb": "2097148", 2023-12-10T19:31:08.020 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "mem_total_kb": "32746948", 2023-12-10T19:31:08.020 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "os": "Linux" 2023-12-10T19:31:08.020 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: }, 2023-12-10T19:31:08.020 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "task_status": {} 2023-12-10T19:31:08.020 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: }, 2023-12-10T19:31:08.020 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "6167": { 2023-12-10T19:31:08.021 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "start_epoch": 42, 2023-12-10T19:31:08.021 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "start_stamp": "2023-12-10T19:28:13.004676+0000", 2023-12-10T19:31:08.021 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "gid": 6167, 2023-12-10T19:31:08.021 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "addr": "172.21.15.76:0/1534972917", 2023-12-10T19:31:08.021 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "metadata": { 2023-12-10T19:31:08.021 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "arch": "x86_64", 2023-12-10T19:31:08.021 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "ceph_release": "reef", 2023-12-10T19:31:08.021 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "ceph_version": "ceph version 19.0.0-130-g3d2db44f (3d2db44fa991bc355074aa9b7647f63b77201fef) reef (dev)", 2023-12-10T19:31:08.021 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "ceph_version_short": "19.0.0-130-g3d2db44f", 2023-12-10T19:31:08.021 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2023-12-10T19:31:08.021 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "distro": "ubuntu", 2023-12-10T19:31:08.021 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2023-12-10T19:31:08.021 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "distro_version": "22.04", 2023-12-10T19:31:08.021 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "hostname": "smithi076", 2023-12-10T19:31:08.021 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "id": "mirror.1", 2023-12-10T19:31:08.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "instance_id": "6167", 2023-12-10T19:31:08.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "kernel_description": "#101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023", 2023-12-10T19:31:08.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "kernel_version": "5.15.0-91-generic", 2023-12-10T19:31:08.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "mem_swap_kb": "2097148", 2023-12-10T19:31:08.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "mem_total_kb": "32746948", 2023-12-10T19:31:08.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "os": "Linux" 2023-12-10T19:31:08.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: }, 2023-12-10T19:31:08.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "task_status": {} 2023-12-10T19:31:08.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: } 2023-12-10T19:31:08.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: } 2023-12-10T19:31:08.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: } 2023-12-10T19:31:08.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: } 2023-12-10T19:31:08.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:} 2023-12-10T19:31:08.032 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ CEPH_ARGS= ceph --cluster cluster1 service status 2023-12-10T19:31:08.396 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:{ 2023-12-10T19:31:08.396 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "rbd-mirror": { 2023-12-10T19:31:08.396 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "5246": { 2023-12-10T19:31:08.396 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "status_stamp": "2023-12-10T19:31:06.016991+0000", 2023-12-10T19:31:08.396 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "last_beacon": "2023-12-10T19:31:06.016991+0000", 2023-12-10T19:31:08.396 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "status": { 2023-12-10T19:31:08.397 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "json": "{\"3\":{\"name\":\"mirror\",\"callouts\":{},\"image_assigned_count\":1,\"image_error_count\":0,\"image_warning_count\":0,\"instance_id\":\"5249\",\"namespaces\":{\"ns1\":{\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0},\"ns2\":{\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0}}},\"4\":{\"name\":\"mirror_parent\",\"callouts\":{},\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0,\"instance_id\":\"5255\"}}" 2023-12-10T19:31:08.397 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: } 2023-12-10T19:31:08.397 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: }, 2023-12-10T19:31:08.397 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "6041": { 2023-12-10T19:31:08.397 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "status_stamp": "2023-12-10T19:30:51.021129+0000", 2023-12-10T19:31:08.397 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "last_beacon": "2023-12-10T19:31:06.021866+0000", 2023-12-10T19:31:08.397 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "status": { 2023-12-10T19:31:08.397 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "json": "{\"3\":{\"name\":\"mirror\",\"callouts\":{},\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0,\"instance_id\":\"6044\",\"namespaces\":{\"ns1\":{\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0},\"ns2\":{\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0}}},\"4\":{\"name\":\"mirror_parent\",\"callouts\":{},\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0,\"instance_id\":\"6047\"}}" 2023-12-10T19:31:08.397 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: } 2023-12-10T19:31:08.397 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: }, 2023-12-10T19:31:08.397 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "6082": { 2023-12-10T19:31:08.397 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "status_stamp": "2023-12-10T19:30:57.045296+0000", 2023-12-10T19:31:08.397 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "last_beacon": "2023-12-10T19:31:07.045762+0000", 2023-12-10T19:31:08.397 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "status": { 2023-12-10T19:31:08.397 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "json": "{\"3\":{\"name\":\"mirror\",\"callouts\":{},\"image_assigned_count\":0,\"image_error_count\":0,\"image_local_count\":0,\"image_remote_count\":1,\"image_warning_count\":0,\"instance_id\":\"6085\",\"leader\":true,\"namespaces\":{\"ns1\":{\"image_assigned_count\":0,\"image_error_count\":0,\"image_local_count\":0,\"image_remote_count\":0,\"image_warning_count\":0},\"ns2\":{\"image_assigned_count\":0,\"image_error_count\":0,\"image_local_count\":0,\"image_remote_count\":0,\"image_warning_count\":0}}},\"4\":{\"name\":\"mirror_parent\",\"callouts\":{},\"image_assigned_count\":0,\"image_error_count\":0,\"image_local_count\":0,\"image_remote_count\":0,\"image_warning_count\":0,\"instance_id\":\"6088\",\"leader\":true}}" 2023-12-10T19:31:08.398 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: } 2023-12-10T19:31:08.398 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: }, 2023-12-10T19:31:08.398 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "6167": { 2023-12-10T19:31:08.398 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "status_stamp": "2023-12-10T19:30:58.013316+0000", 2023-12-10T19:31:08.398 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "last_beacon": "2023-12-10T19:31:08.013775+0000", 2023-12-10T19:31:08.398 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "status": { 2023-12-10T19:31:08.398 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: "json": "{\"3\":{\"name\":\"mirror\",\"callouts\":{},\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0,\"instance_id\":\"6170\",\"namespaces\":{\"ns1\":{\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0}}},\"4\":{\"name\":\"mirror_parent\",\"callouts\":{},\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0,\"instance_id\":\"6173\"}}" 2023-12-10T19:31:08.398 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: } 2023-12-10T19:31:08.398 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: } 2023-12-10T19:31:08.398 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: } 2023-12-10T19:31:08.398 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:} 2023-12-10T19:31:08.412 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster1 mirror images 2023-12-10T19:31:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo 2023-12-10T19:31:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1 mirror images 2023-12-10T19:31:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror --namespace ls -l 2023-12-10T19:31:08.460 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:08.461 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster1 mirror mirror pool info 2023-12-10T19:31:08.461 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo 2023-12-10T19:31:08.461 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1 mirror mirror pool info 2023-12-10T19:31:08.461 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror --namespace mirror pool info 2023-12-10T19:31:08.503 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mode: image 2023-12-10T19:31:08.503 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Site Name: cluster1 2023-12-10T19:31:08.503 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:08.503 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Peer Sites: 2023-12-10T19:31:08.503 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:08.503 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:UUID: cea0e0df-3ba5-45fb-ab55-c5b809a43d2b 2023-12-10T19:31:08.503 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Name: cluster2 2023-12-10T19:31:08.503 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirror UUID: f4a845a0-65d8-4baa-abd4-fd2893f9af83 2023-12-10T19:31:08.503 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Direction: rx-tx 2023-12-10T19:31:08.504 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Client: client.mirror 2023-12-10T19:31:08.506 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:08.506 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster1 mirror mirror pool status 2023-12-10T19:31:08.506 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo 2023-12-10T19:31:08.506 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1 mirror mirror pool status 2023-12-10T19:31:08.506 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ CEPH_ARGS= rbd --cluster cluster1 -p mirror --namespace mirror pool status --verbose 2023-12-10T19:31:08.550 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:health: OK 2023-12-10T19:31:08.550 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:daemon health: OK 2023-12-10T19:31:08.550 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:image health: OK 2023-12-10T19:31:08.550 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:images: 0 total 2023-12-10T19:31:08.550 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:08.550 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:DAEMONS 2023-12-10T19:31:08.550 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:service 5246: 2023-12-10T19:31:08.550 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: instance_id: 5249 2023-12-10T19:31:08.550 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: client_id: mirror.2 2023-12-10T19:31:08.550 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: hostname: smithi076 2023-12-10T19:31:08.551 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: version: 19.0.0-130-g3d2db44f 2023-12-10T19:31:08.551 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: leader: false 2023-12-10T19:31:08.551 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: health: OK 2023-12-10T19:31:08.551 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:08.551 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:service 6041: 2023-12-10T19:31:08.551 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: instance_id: 6044 2023-12-10T19:31:08.551 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: client_id: mirror.0 2023-12-10T19:31:08.551 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: hostname: smithi076 2023-12-10T19:31:08.551 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: version: 19.0.0-130-g3d2db44f 2023-12-10T19:31:08.551 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: leader: false 2023-12-10T19:31:08.551 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: health: OK 2023-12-10T19:31:08.551 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:08.551 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:service 6082: 2023-12-10T19:31:08.551 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: instance_id: 6085 2023-12-10T19:31:08.551 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: client_id: mirror.3 2023-12-10T19:31:08.551 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: hostname: smithi076 2023-12-10T19:31:08.551 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: version: 19.0.0-130-g3d2db44f 2023-12-10T19:31:08.552 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: leader: true 2023-12-10T19:31:08.552 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: health: OK 2023-12-10T19:31:08.552 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:08.552 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:service 6167: 2023-12-10T19:31:08.552 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: instance_id: 6170 2023-12-10T19:31:08.552 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: client_id: mirror.1 2023-12-10T19:31:08.552 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: hostname: smithi076 2023-12-10T19:31:08.552 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: version: 19.0.0-130-g3d2db44f 2023-12-10T19:31:08.552 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: leader: false 2023-12-10T19:31:08.552 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: health: OK 2023-12-10T19:31:08.552 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:08.552 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:08.561 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo 2023-12-10T19:31:08.561 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:IMAGES 2023-12-10T19:31:08.562 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror --namespace ls 2023-12-10T19:31:08.602 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster1 mirror rbd_mirroring omap vals 2023-12-10T19:31:08.603 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1 mirror rbd_mirroring omap vals 2023-12-10T19:31:08.603 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rados --cluster cluster1 -p mirror --namespace listomapvals rbd_mirroring 2023-12-10T19:31:08.644 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:mirror_mode 2023-12-10T19:31:08.644 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:value (4 bytes) : 2023-12-10T19:31:08.644 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:00000000 01 00 00 00 |....| 2023-12-10T19:31:08.644 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:00000004 2023-12-10T19:31:08.644 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:08.644 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:mirror_peer_cea0e0df-3ba5-45fb-ab55-c5b809a43d2b 2023-12-10T19:31:08.644 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:value (132 bytes) : 2023-12-10T19:31:08.644 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:00000000 02 01 7e 00 00 00 24 00 00 00 63 65 61 30 65 30 |..~...$...cea0e0| 2023-12-10T19:31:08.644 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:00000010 64 66 2d 33 62 61 35 2d 34 35 66 62 2d 61 62 35 |df-3ba5-45fb-ab5| 2023-12-10T19:31:08.645 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:00000020 35 2d 63 35 62 38 30 39 61 34 33 64 32 62 08 00 |5-c5b809a43d2b..| 2023-12-10T19:31:08.645 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:00000030 00 00 63 6c 75 73 74 65 72 32 0d 00 00 00 63 6c |..cluster2....cl| 2023-12-10T19:31:08.645 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:00000040 69 65 6e 74 2e 6d 69 72 72 6f 72 ff ff ff ff ff |ient.mirror.....| 2023-12-10T19:31:08.645 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:00000050 ff ff ff 02 24 00 00 00 66 34 61 38 34 35 61 30 |....$...f4a845a0| 2023-12-10T19:31:08.645 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:00000060 2d 36 35 64 38 2d 34 62 61 61 2d 61 62 64 34 2d |-65d8-4baa-abd4-| 2023-12-10T19:31:08.645 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:00000070 66 64 32 38 39 33 66 39 61 66 38 33 fb 11 76 65 |fd2893f9af83..ve| 2023-12-10T19:31:08.645 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:00000080 be 5b 8f 15 |.[..| 2023-12-10T19:31:08.645 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:00000084 2023-12-10T19:31:08.645 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:08.645 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:mirror_uuid 2023-12-10T19:31:08.645 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:value (36 bytes) : 2023-12-10T19:31:08.645 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:00000000 39 65 38 64 32 34 61 65 2d 66 30 61 34 2d 34 35 |9e8d24ae-f0a4-45| 2023-12-10T19:31:08.645 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:00000010 62 66 2d 61 33 63 38 2d 35 34 32 36 65 38 39 66 |bf-a3c8-5426e89f| 2023-12-10T19:31:08.645 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:00000020 65 61 66 32 |eaf2| 2023-12-10T19:31:08.645 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:00000024 2023-12-10T19:31:08.645 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:08.646 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1 mirror rbd_mirror_leader omap vals 2023-12-10T19:31:08.646 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster1 mirror rbd_mirror_leader omap vals 2023-12-10T19:31:08.647 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rados --cluster cluster1 -p mirror --namespace listomapvals rbd_mirror_leader 2023-12-10T19:31:08.682 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:image_map_f22ea0ff-01dc-4ad2-9137-754811a1953d 2023-12-10T19:31:08.683 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:value (36 bytes) : 2023-12-10T19:31:08.683 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 35 32 34 39 67 11 |..........5249g.| 2023-12-10T19:31:08.683 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:00000010 76 65 ae 70 e2 05 0a 00 00 00 01 01 04 00 00 00 |ve.p............| 2023-12-10T19:31:08.683 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:00000020 00 00 00 00 |....| 2023-12-10T19:31:08.683 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:00000024 2023-12-10T19:31:08.683 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:08.683 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:instance_5249 2023-12-10T19:31:08.683 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:value (0 bytes) : 2023-12-10T19:31:08.683 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:08.683 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:instance_6044 2023-12-10T19:31:08.683 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:value (0 bytes) : 2023-12-10T19:31:08.683 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:08.683 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:instance_6085 2023-12-10T19:31:08.684 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:value (0 bytes) : 2023-12-10T19:31:08.684 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:08.684 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:instance_6170 2023-12-10T19:31:08.684 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:value (0 bytes) : 2023-12-10T19:31:08.684 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:08.685 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo 2023-12-10T19:31:08.685 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:08.685 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster1 mirror ns1 images 2023-12-10T19:31:08.685 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1 mirror ns1 images 2023-12-10T19:31:08.685 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror --namespace ns1 ls -l 2023-12-10T19:31:08.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:08.727 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster1 mirrorns1 mirror pool info 2023-12-10T19:31:08.728 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo 2023-12-10T19:31:08.728 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1 mirrorns1 mirror pool info 2023-12-10T19:31:08.728 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror --namespace ns1 mirror pool info 2023-12-10T19:31:08.766 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mode: image 2023-12-10T19:31:08.769 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:08.769 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster1 mirrorns1 mirror pool status 2023-12-10T19:31:08.769 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo 2023-12-10T19:31:08.769 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1 mirrorns1 mirror pool status 2023-12-10T19:31:08.769 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ CEPH_ARGS= rbd --cluster cluster1 -p mirror --namespace ns1 mirror pool status --verbose 2023-12-10T19:31:08.810 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:health: OK 2023-12-10T19:31:08.810 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:daemon health: OK 2023-12-10T19:31:08.810 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:image health: OK 2023-12-10T19:31:08.810 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:images: 0 total 2023-12-10T19:31:08.810 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:08.810 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:DAEMONS 2023-12-10T19:31:08.810 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:service 5246: 2023-12-10T19:31:08.810 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: instance_id: 5249 2023-12-10T19:31:08.810 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: client_id: mirror.2 2023-12-10T19:31:08.810 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: hostname: smithi076 2023-12-10T19:31:08.810 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: version: 19.0.0-130-g3d2db44f 2023-12-10T19:31:08.810 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: leader: false 2023-12-10T19:31:08.811 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: health: OK 2023-12-10T19:31:08.811 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:08.811 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:service 6041: 2023-12-10T19:31:08.811 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: instance_id: 6044 2023-12-10T19:31:08.811 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: client_id: mirror.0 2023-12-10T19:31:08.811 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: hostname: smithi076 2023-12-10T19:31:08.811 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: version: 19.0.0-130-g3d2db44f 2023-12-10T19:31:08.811 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: leader: false 2023-12-10T19:31:08.811 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: health: OK 2023-12-10T19:31:08.811 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:08.811 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:service 6082: 2023-12-10T19:31:08.811 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: instance_id: 6085 2023-12-10T19:31:08.811 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: client_id: mirror.3 2023-12-10T19:31:08.811 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: hostname: smithi076 2023-12-10T19:31:08.811 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: version: 19.0.0-130-g3d2db44f 2023-12-10T19:31:08.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: leader: true 2023-12-10T19:31:08.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: health: OK 2023-12-10T19:31:08.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:08.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:service 6167: 2023-12-10T19:31:08.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: instance_id: 6170 2023-12-10T19:31:08.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: client_id: mirror.1 2023-12-10T19:31:08.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: hostname: smithi076 2023-12-10T19:31:08.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: version: 19.0.0-130-g3d2db44f 2023-12-10T19:31:08.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: leader: false 2023-12-10T19:31:08.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: health: OK 2023-12-10T19:31:08.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:08.812 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:08.817 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:IMAGES 2023-12-10T19:31:08.818 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo 2023-12-10T19:31:08.818 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror --namespace ns1 ls 2023-12-10T19:31:08.857 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster1 mirror ns1 rbd_mirroring omap vals 2023-12-10T19:31:08.857 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1 mirror ns1 rbd_mirroring omap vals 2023-12-10T19:31:08.857 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rados --cluster cluster1 -p mirror --namespace ns1 listomapvals rbd_mirroring 2023-12-10T19:31:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:mirror_mode 2023-12-10T19:31:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:value (4 bytes) : 2023-12-10T19:31:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:00000000 01 00 00 00 |....| 2023-12-10T19:31:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:00000004 2023-12-10T19:31:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:mirror_uuid 2023-12-10T19:31:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:value (36 bytes) : 2023-12-10T19:31:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:00000000 36 38 64 61 63 63 64 63 2d 66 36 30 64 2d 34 38 |68daccdc-f60d-48| 2023-12-10T19:31:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:00000010 31 30 2d 38 30 36 31 2d 39 31 32 31 66 39 36 36 |10-8061-9121f966| 2023-12-10T19:31:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:00000020 31 61 61 32 |1aa2| 2023-12-10T19:31:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:00000024 2023-12-10T19:31:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:08.894 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1 mirror ns1 rbd_mirror_leader omap vals 2023-12-10T19:31:08.894 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster1 mirror ns1 rbd_mirror_leader omap vals 2023-12-10T19:31:08.895 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rados --cluster cluster1 -p mirror --namespace ns1 listomapvals rbd_mirror_leader 2023-12-10T19:31:08.930 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:instance_5249 2023-12-10T19:31:08.930 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:value (0 bytes) : 2023-12-10T19:31:08.930 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:08.930 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:instance_6044 2023-12-10T19:31:08.930 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:value (0 bytes) : 2023-12-10T19:31:08.930 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:08.930 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:instance_6085 2023-12-10T19:31:08.930 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:value (0 bytes) : 2023-12-10T19:31:08.930 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:08.931 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:instance_6170 2023-12-10T19:31:08.931 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:value (0 bytes) : 2023-12-10T19:31:08.931 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:08.932 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo 2023-12-10T19:31:08.932 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:08.932 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster1 mirror ns2 images 2023-12-10T19:31:08.932 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1 mirror ns2 images 2023-12-10T19:31:08.932 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror --namespace ns2 ls -l 2023-12-10T19:31:08.975 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:08.975 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster1 mirrorns2 mirror pool info 2023-12-10T19:31:08.975 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo 2023-12-10T19:31:08.975 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1 mirrorns2 mirror pool info 2023-12-10T19:31:08.975 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror --namespace ns2 mirror pool info 2023-12-10T19:31:09.015 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mode: image 2023-12-10T19:31:09.017 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo 2023-12-10T19:31:09.018 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:09.018 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster1 mirrorns2 mirror pool status 2023-12-10T19:31:09.018 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1 mirrorns2 mirror pool status 2023-12-10T19:31:09.018 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ CEPH_ARGS= rbd --cluster cluster1 -p mirror --namespace ns2 mirror pool status --verbose 2023-12-10T19:31:09.062 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:health: OK 2023-12-10T19:31:09.086 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:daemon health: OK 2023-12-10T19:31:09.086 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:image health: OK 2023-12-10T19:31:09.086 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:images: 0 total 2023-12-10T19:31:09.086 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:09.086 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:DAEMONS 2023-12-10T19:31:09.086 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:service 5246: 2023-12-10T19:31:09.086 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: instance_id: 5249 2023-12-10T19:31:09.087 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: client_id: mirror.2 2023-12-10T19:31:09.087 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: hostname: smithi076 2023-12-10T19:31:09.087 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: version: 19.0.0-130-g3d2db44f 2023-12-10T19:31:09.087 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: leader: false 2023-12-10T19:31:09.087 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: health: OK 2023-12-10T19:31:09.087 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:09.087 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:service 6041: 2023-12-10T19:31:09.087 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: instance_id: 6044 2023-12-10T19:31:09.087 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: client_id: mirror.0 2023-12-10T19:31:09.087 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: hostname: smithi076 2023-12-10T19:31:09.087 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: version: 19.0.0-130-g3d2db44f 2023-12-10T19:31:09.087 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: leader: false 2023-12-10T19:31:09.087 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: health: OK 2023-12-10T19:31:09.087 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:09.087 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:service 6082: 2023-12-10T19:31:09.087 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: instance_id: 6085 2023-12-10T19:31:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: client_id: mirror.3 2023-12-10T19:31:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: hostname: smithi076 2023-12-10T19:31:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: version: 19.0.0-130-g3d2db44f 2023-12-10T19:31:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: leader: true 2023-12-10T19:31:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: health: OK 2023-12-10T19:31:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:service 6167: 2023-12-10T19:31:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: instance_id: 6170 2023-12-10T19:31:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: client_id: mirror.1 2023-12-10T19:31:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: hostname: smithi076 2023-12-10T19:31:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: version: 19.0.0-130-g3d2db44f 2023-12-10T19:31:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: leader: false 2023-12-10T19:31:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: health: OK 2023-12-10T19:31:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:09.089 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo 2023-12-10T19:31:09.089 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:IMAGES 2023-12-10T19:31:09.090 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror --namespace ns2 ls 2023-12-10T19:31:09.110 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster1 mirror ns2 rbd_mirroring omap vals 2023-12-10T19:31:09.111 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1 mirror ns2 rbd_mirroring omap vals 2023-12-10T19:31:09.111 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rados --cluster cluster1 -p mirror --namespace ns2 listomapvals rbd_mirroring 2023-12-10T19:31:09.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:mirror_mode 2023-12-10T19:31:09.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:value (4 bytes) : 2023-12-10T19:31:09.148 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:00000000 01 00 00 00 |....| 2023-12-10T19:31:09.149 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:00000004 2023-12-10T19:31:09.149 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:09.149 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:mirror_uuid 2023-12-10T19:31:09.149 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:value (36 bytes) : 2023-12-10T19:31:09.149 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:00000000 62 32 37 65 30 39 66 38 2d 65 36 61 37 2d 34 65 |b27e09f8-e6a7-4e| 2023-12-10T19:31:09.149 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:00000010 38 30 2d 38 37 61 33 2d 63 37 34 39 65 35 63 62 |80-87a3-c749e5cb| 2023-12-10T19:31:09.149 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:00000020 31 38 39 36 |1896| 2023-12-10T19:31:09.149 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:00000024 2023-12-10T19:31:09.149 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:09.151 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster1 mirror ns2 rbd_mirror_leader omap vals 2023-12-10T19:31:09.151 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1 mirror ns2 rbd_mirror_leader omap vals 2023-12-10T19:31:09.151 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rados --cluster cluster1 -p mirror --namespace ns2 listomapvals rbd_mirror_leader 2023-12-10T19:31:09.186 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:instance_5249 2023-12-10T19:31:09.186 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:value (0 bytes) : 2023-12-10T19:31:09.186 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:09.186 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:instance_6044 2023-12-10T19:31:09.186 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:value (0 bytes) : 2023-12-10T19:31:09.186 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:09.186 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:instance_6085 2023-12-10T19:31:09.187 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:value (0 bytes) : 2023-12-10T19:31:09.187 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:09.188 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo 2023-12-10T19:31:09.189 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:09.189 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster1 mirror_parent images 2023-12-10T19:31:09.189 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1 mirror_parent images 2023-12-10T19:31:09.189 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace ls -l 2023-12-10T19:31:09.226 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:09.227 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster1 mirror_parent mirror pool info 2023-12-10T19:31:09.227 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo 2023-12-10T19:31:09.227 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1 mirror_parent mirror pool info 2023-12-10T19:31:09.227 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace mirror pool info 2023-12-10T19:31:09.264 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mode: image 2023-12-10T19:31:09.264 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Site Name: cluster1 2023-12-10T19:31:09.265 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:09.265 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Peer Sites: 2023-12-10T19:31:09.265 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:09.265 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:UUID: 0f54ea86-1117-4a80-9b51-0eebdfe1047a 2023-12-10T19:31:09.265 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Name: cluster2 2023-12-10T19:31:09.265 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Mirror UUID: 41705799-d226-4fd0-8da2-b3037b254050 2023-12-10T19:31:09.265 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Direction: rx-tx 2023-12-10T19:31:09.266 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:Client: client.mirror 2023-12-10T19:31:09.267 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:09.267 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster1 mirror_parent mirror pool status 2023-12-10T19:31:09.267 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo 2023-12-10T19:31:09.268 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1 mirror_parent mirror pool status 2023-12-10T19:31:09.268 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ CEPH_ARGS= rbd --cluster cluster1 -p mirror_parent --namespace mirror pool status --verbose 2023-12-10T19:31:09.308 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:health: OK 2023-12-10T19:31:09.308 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:daemon health: OK 2023-12-10T19:31:09.308 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:image health: OK 2023-12-10T19:31:09.309 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:images: 0 total 2023-12-10T19:31:09.309 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:09.309 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:DAEMONS 2023-12-10T19:31:09.309 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:service 5246: 2023-12-10T19:31:09.309 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: instance_id: 5255 2023-12-10T19:31:09.309 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: client_id: mirror.2 2023-12-10T19:31:09.309 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: hostname: smithi076 2023-12-10T19:31:09.309 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: version: 19.0.0-130-g3d2db44f 2023-12-10T19:31:09.309 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: leader: false 2023-12-10T19:31:09.309 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: health: OK 2023-12-10T19:31:09.309 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:09.309 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:service 6041: 2023-12-10T19:31:09.309 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: instance_id: 6047 2023-12-10T19:31:09.309 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: client_id: mirror.0 2023-12-10T19:31:09.309 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: hostname: smithi076 2023-12-10T19:31:09.309 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: version: 19.0.0-130-g3d2db44f 2023-12-10T19:31:09.310 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: leader: false 2023-12-10T19:31:09.310 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: health: OK 2023-12-10T19:31:09.310 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:09.310 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:service 6082: 2023-12-10T19:31:09.310 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: instance_id: 6088 2023-12-10T19:31:09.310 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: client_id: mirror.3 2023-12-10T19:31:09.310 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: hostname: smithi076 2023-12-10T19:31:09.310 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: version: 19.0.0-130-g3d2db44f 2023-12-10T19:31:09.310 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: leader: true 2023-12-10T19:31:09.310 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: health: OK 2023-12-10T19:31:09.310 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:09.310 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:service 6167: 2023-12-10T19:31:09.310 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: instance_id: 6173 2023-12-10T19:31:09.310 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: client_id: mirror.1 2023-12-10T19:31:09.310 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: hostname: smithi076 2023-12-10T19:31:09.310 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: version: 19.0.0-130-g3d2db44f 2023-12-10T19:31:09.311 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: leader: false 2023-12-10T19:31:09.311 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: health: OK 2023-12-10T19:31:09.311 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:09.311 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:09.315 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:IMAGES 2023-12-10T19:31:09.316 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo 2023-12-10T19:31:09.316 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace ls 2023-12-10T19:31:09.353 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster1 mirror_parent rbd_mirroring omap vals 2023-12-10T19:31:09.354 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1 mirror_parent rbd_mirroring omap vals 2023-12-10T19:31:09.354 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rados --cluster cluster1 -p mirror_parent --namespace listomapvals rbd_mirroring 2023-12-10T19:31:09.389 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:mirror_mode 2023-12-10T19:31:09.389 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:value (4 bytes) : 2023-12-10T19:31:09.389 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:00000000 01 00 00 00 |....| 2023-12-10T19:31:09.389 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:00000004 2023-12-10T19:31:09.390 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:09.390 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:mirror_peer_0f54ea86-1117-4a80-9b51-0eebdfe1047a 2023-12-10T19:31:09.390 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:value (132 bytes) : 2023-12-10T19:31:09.390 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:00000000 02 01 7e 00 00 00 24 00 00 00 30 66 35 34 65 61 |..~...$...0f54ea| 2023-12-10T19:31:09.390 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:00000010 38 36 2d 31 31 31 37 2d 34 61 38 30 2d 39 62 35 |86-1117-4a80-9b5| 2023-12-10T19:31:09.390 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:00000020 31 2d 30 65 65 62 64 66 65 31 30 34 37 61 08 00 |1-0eebdfe1047a..| 2023-12-10T19:31:09.390 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:00000030 00 00 63 6c 75 73 74 65 72 32 0d 00 00 00 63 6c |..cluster2....cl| 2023-12-10T19:31:09.390 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:00000040 69 65 6e 74 2e 6d 69 72 72 6f 72 ff ff ff ff ff |ient.mirror.....| 2023-12-10T19:31:09.390 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:00000050 ff ff ff 02 24 00 00 00 34 31 37 30 35 37 39 39 |....$...41705799| 2023-12-10T19:31:09.390 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:00000060 2d 64 32 32 36 2d 34 66 64 30 2d 38 64 61 32 2d |-d226-4fd0-8da2-| 2023-12-10T19:31:09.390 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:00000070 62 33 30 33 37 62 32 35 34 30 35 30 fb 11 76 65 |b3037b254050..ve| 2023-12-10T19:31:09.390 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:00000080 d0 0b b0 13 |....| 2023-12-10T19:31:09.390 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:00000084 2023-12-10T19:31:09.390 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:mirror_uuid 2023-12-10T19:31:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:value (36 bytes) : 2023-12-10T19:31:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:00000000 61 30 37 61 63 38 61 65 2d 37 32 66 38 2d 34 31 |a07ac8ae-72f8-41| 2023-12-10T19:31:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:00000010 64 32 2d 61 32 64 64 2d 35 39 62 63 34 63 65 62 |d2-a2dd-59bc4ceb| 2023-12-10T19:31:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:00000020 62 37 38 39 |b789| 2023-12-10T19:31:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:00000024 2023-12-10T19:31:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1 mirror_parent rbd_mirror_leader omap vals 2023-12-10T19:31:09.392 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster1 mirror_parent rbd_mirror_leader omap vals 2023-12-10T19:31:09.392 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rados --cluster cluster1 -p mirror_parent --namespace listomapvals rbd_mirror_leader 2023-12-10T19:31:09.426 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:instance_5255 2023-12-10T19:31:09.426 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:value (0 bytes) : 2023-12-10T19:31:09.426 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:09.426 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:instance_6047 2023-12-10T19:31:09.426 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:value (0 bytes) : 2023-12-10T19:31:09.427 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:09.427 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:instance_6088 2023-12-10T19:31:09.427 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:value (0 bytes) : 2023-12-10T19:31:09.427 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:09.427 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:instance_6173 2023-12-10T19:31:09.427 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:value (0 bytes) : 2023-12-10T19:31:09.427 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:09.429 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:09.429 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo 2023-12-10T19:31:09.429 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster1 mirror_parent ns1 images 2023-12-10T19:31:09.429 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1 mirror_parent ns1 images 2023-12-10T19:31:09.430 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace ns1 ls -l 2023-12-10T19:31:09.465 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: namespace 'ns1' does not exist. 2023-12-10T19:31:09.466 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: listing images failed: (2) No such file or directory 2023-12-10T19:31:09.468 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:09.468 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo 2023-12-10T19:31:09.468 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1 mirror_parentns1 mirror pool info 2023-12-10T19:31:09.469 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster1 mirror_parentns1 mirror pool info 2023-12-10T19:31:09.469 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace ns1 mirror pool info 2023-12-10T19:31:09.503 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: namespace 'ns1' does not exist. 2023-12-10T19:31:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo 2023-12-10T19:31:09.509 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:09.510 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster1 mirror_parentns1 mirror pool status 2023-12-10T19:31:09.510 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1 mirror_parentns1 mirror pool status 2023-12-10T19:31:09.510 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ CEPH_ARGS= rbd --cluster cluster1 -p mirror_parent --namespace ns1 mirror pool status --verbose 2023-12-10T19:31:09.543 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: namespace 'ns1' does not exist. 2023-12-10T19:31:09.546 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:09.547 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo 2023-12-10T19:31:09.547 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace ns1 ls 2023-12-10T19:31:09.584 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster1 mirror_parent ns1 rbd_mirroring omap vals 2023-12-10T19:31:09.584 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1 mirror_parent ns1 rbd_mirroring omap vals 2023-12-10T19:31:09.584 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rados --cluster cluster1 -p mirror_parent --namespace ns1 listomapvals rbd_mirroring 2023-12-10T19:31:09.619 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:error getting omap keys mirror_parent/rbd_mirroring: (2) No such file or directory 2023-12-10T19:31:09.621 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1 mirror_parent ns1 rbd_mirror_leader omap vals 2023-12-10T19:31:09.621 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster1 mirror_parent ns1 rbd_mirror_leader omap vals 2023-12-10T19:31:09.622 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rados --cluster cluster1 -p mirror_parent --namespace ns1 listomapvals rbd_mirror_leader 2023-12-10T19:31:09.657 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:error getting omap keys mirror_parent/rbd_mirror_leader: (2) No such file or directory 2023-12-10T19:31:09.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:09.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo 2023-12-10T19:31:09.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1 mirror_parent ns2 images 2023-12-10T19:31:09.660 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster1 mirror_parent ns2 images 2023-12-10T19:31:09.661 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace ns2 ls -l 2023-12-10T19:31:09.695 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: namespace 'ns2' does not exist. 2023-12-10T19:31:09.695 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: listing images failed: (2) No such file or directory 2023-12-10T19:31:09.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:09.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo 2023-12-10T19:31:09.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1 mirror_parentns2 mirror pool info 2023-12-10T19:31:09.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster1 mirror_parentns2 mirror pool info 2023-12-10T19:31:09.698 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace ns2 mirror pool info 2023-12-10T19:31:09.734 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: namespace 'ns2' does not exist. 2023-12-10T19:31:09.736 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:09.736 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster1 mirror_parentns2 mirror pool status 2023-12-10T19:31:09.736 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo 2023-12-10T19:31:09.736 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1 mirror_parentns2 mirror pool status 2023-12-10T19:31:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ CEPH_ARGS= rbd --cluster cluster1 -p mirror_parent --namespace ns2 mirror pool status --verbose 2023-12-10T19:31:09.771 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: namespace 'ns2' does not exist. 2023-12-10T19:31:09.774 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:09.774 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo 2023-12-10T19:31:09.774 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace ns2 ls 2023-12-10T19:31:09.810 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster1 mirror_parent ns2 rbd_mirroring omap vals 2023-12-10T19:31:09.810 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1 mirror_parent ns2 rbd_mirroring omap vals 2023-12-10T19:31:09.810 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rados --cluster cluster1 -p mirror_parent --namespace ns2 listomapvals rbd_mirroring 2023-12-10T19:31:09.844 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:error getting omap keys mirror_parent/rbd_mirroring: (2) No such file or directory 2023-12-10T19:31:09.847 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster1 mirror_parent ns2 rbd_mirror_leader omap vals 2023-12-10T19:31:09.847 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1 mirror_parent ns2 rbd_mirror_leader omap vals 2023-12-10T19:31:09.847 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rados --cluster cluster1 -p mirror_parent --namespace ns2 listomapvals rbd_mirror_leader 2023-12-10T19:31:09.881 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:error getting omap keys mirror_parent/rbd_mirror_leader: (2) No such file or directory 2023-12-10T19:31:09.883 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo 2023-12-10T19:31:09.884 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:31:09.884 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster2 status 2023-12-10T19:31:09.884 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2 status 2023-12-10T19:31:09.884 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ CEPH_ARGS= ceph --cluster cluster2 -s 2023-12-10T19:31:22.909 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-12-10T19:31:22.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-12-10T19:31:22.911 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2023-12-10T19:31:22.911 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-12-10T19:31:22.911 INFO:teuthology.orchestra.run:waiting for 600 2023-12-10T19:31:22.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr:2023-12-10T19:31:22.915+0000 7f1b80bf0640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 72689) UID: 1000 2023-12-10T19:36:09.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:2023-12-10T19:36:09.975+0000 7f15f4cf8640 0 monclient(hunting): authenticate timed out after 300 2023-12-10T19:36:09.977 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2023-12-10T19:36:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ CEPH_ARGS= ceph --cluster cluster2 service dump 2023-12-10T19:38:57.058 INFO:tasks.rbd_mirror_thrash:Failed to stop cluster2.client.mirror.1 2023-12-10T19:38:57.059 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 6 2023-12-10T19:38:57.060 ERROR:tasks.rbd_mirror_thrash:exception: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/rbd_mirror_thrash.py", line 84, in _run self.do_thrash() File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/rbd_mirror_thrash.py", line 156, in do_thrash run.wait([daemon.proc], timeout=600) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (101) after waiting for 600 seconds 2023-12-10T19:38:57.061 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr:*** Caught signal (Aborted) ** 2023-12-10T19:38:57.061 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: in thread 7f2c64e27880 thread_name:rbd-mirror 2023-12-10T19:38:57.063 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: ceph version 19.0.0-130-g3d2db44f (3d2db44fa991bc355074aa9b7647f63b77201fef) reef (dev) 2023-12-10T19:38:57.063 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f2c65cba520] 2023-12-10T19:38:57.063 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: 2: /lib/x86_64-linux-gnu/libc.so.6(+0x91197) [0x7f2c65d09197] 2023-12-10T19:38:57.063 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: 3: pthread_cond_wait() 2023-12-10T19:38:57.063 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: 4: /lib/librados.so.2(+0x9be8f) [0x7f2c67324e8f] 2023-12-10T19:38:57.063 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: 5: (librados::v14_2_0::IoCtx::operate(std::__cxx11::basic_string, std::allocator > const&, librados::v14_2_0::ObjectReadOperation*, ceph::buffer::v15_2_0::list*)+0x65) [0x7f2c67308785] 2023-12-10T19:38:57.063 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: 6: (librbd::cls_client::mirror_mode_get(librados::v14_2_0::IoCtx*, cls::rbd::MirrorMode*)+0xa8) [0x5655530e0218] 2023-12-10T19:38:57.063 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: 7: (rbd::mirror::ClusterWatcher::read_pool_peers(std::map >, std::less, std::allocator > > > >*)+0x252) [0x565552bdf912] 2023-12-10T19:38:57.063 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: 8: (rbd::mirror::ClusterWatcher::refresh_pools()+0x78) [0x565552be1488] 2023-12-10T19:38:57.063 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: 9: (rbd::mirror::Mirror::run()+0xb5) [0x565552bb8935] 2023-12-10T19:38:57.063 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: 10: main() 2023-12-10T19:38:57.064 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: 11: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f2c65ca1d90] 2023-12-10T19:38:57.064 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: 12: __libc_start_main() 2023-12-10T19:38:57.064 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: 13: _start() 2023-12-10T19:38:57.064 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr:2023-12-10T19:38:57.062+0000 7f2c64e27880 -1 *** Caught signal (Aborted) ** 2023-12-10T19:38:57.064 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: in thread 7f2c64e27880 thread_name:rbd-mirror 2023-12-10T19:38:57.064 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: 2023-12-10T19:38:57.064 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: ceph version 19.0.0-130-g3d2db44f (3d2db44fa991bc355074aa9b7647f63b77201fef) reef (dev) 2023-12-10T19:38:57.064 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f2c65cba520] 2023-12-10T19:38:57.064 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: 2: /lib/x86_64-linux-gnu/libc.so.6(+0x91197) [0x7f2c65d09197] 2023-12-10T19:38:57.064 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: 3: pthread_cond_wait() 2023-12-10T19:38:57.064 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: 4: /lib/librados.so.2(+0x9be8f) [0x7f2c67324e8f] 2023-12-10T19:38:57.064 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: 5: (librados::v14_2_0::IoCtx::operate(std::__cxx11::basic_string, std::allocator > const&, librados::v14_2_0::ObjectReadOperation*, ceph::buffer::v15_2_0::list*)+0x65) [0x7f2c67308785] 2023-12-10T19:38:57.064 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: 6: (librbd::cls_client::mirror_mode_get(librados::v14_2_0::IoCtx*, cls::rbd::MirrorMode*)+0xa8) [0x5655530e0218] 2023-12-10T19:38:57.064 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: 7: (rbd::mirror::ClusterWatcher::read_pool_peers(std::map >, std::less, std::allocator > > > >*)+0x252) [0x565552bdf912] 2023-12-10T19:38:57.065 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: 8: (rbd::mirror::ClusterWatcher::refresh_pools()+0x78) [0x565552be1488] 2023-12-10T19:38:57.065 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: 9: (rbd::mirror::Mirror::run()+0xb5) [0x565552bb8935] 2023-12-10T19:38:57.065 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: 10: main() 2023-12-10T19:38:57.065 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: 11: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f2c65ca1d90] 2023-12-10T19:38:57.065 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: 12: __libc_start_main() 2023-12-10T19:38:57.065 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: 13: _start() 2023-12-10T19:38:57.065 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-10T19:38:57.065 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: 2023-12-10T19:38:57.076 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: -9478> 2023-12-10T19:20:22.076+0000 7f2c5add4640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5655570e2b60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-10T19:38:57.076 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: -9475> 2023-12-10T19:20:22.076+0000 7f2c60de0640 -1 rbd::mirror::ImageReplayer: 0x5655574bcc80 [3/195ff7a6-d9ef-4fb6-afac-e87b92563048] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-10T19:38:57.078 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: -8796> 2023-12-10T19:20:38.083+0000 7f2c5a5d3640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5655570e29c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-10T19:38:57.078 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: -8793> 2023-12-10T19:20:38.083+0000 7f2c615e1640 -1 rbd::mirror::ImageReplayer: 0x5655574bc780 [3/0caf4caf-9ee1-436f-874d-71291f8008fb] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-10T19:38:57.078 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: -8661> 2023-12-10T19:20:46.206+0000 7f2c5a5d3640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56555717e9c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-10T19:38:57.078 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: -8658> 2023-12-10T19:20:46.206+0000 7f2c60de0640 -1 rbd::mirror::ImageReplayer: 0x5655574bd680 [3/40a91520-4e7a-4ba5-aaf0-64312a2dcb5a] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-10T19:38:57.078 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: -8560> 2023-12-10T19:20:48.410+0000 7f2c5add4640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5655570e3520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-10T19:38:57.079 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: -8557> 2023-12-10T19:20:48.410+0000 7f2c615e1640 -1 rbd::mirror::ImageReplayer: 0x565556ff0500 [3/509368f6-035e-4be6-926a-dc4bce662e88] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-10T19:38:57.086 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: -5379> 2023-12-10T19:25:06.040+0000 7f2c5add4640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5655570d8340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-10T19:38:57.086 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: -5376> 2023-12-10T19:25:06.040+0000 7f2c60de0640 -1 rbd::mirror::ImageReplayer: 0x5655574bd400 [3/06e26537-406b-48fb-96a9-75ee5f7af95c] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-10T19:38:57.089 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: -3893> 2023-12-10T19:26:37.049+0000 7f2c5add4640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x565556fcb520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-10T19:38:57.089 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: -3890> 2023-12-10T19:26:37.049+0000 7f2c615e1640 -1 rbd::mirror::ImageReplayer: 0x565556ff0500 [3/674cd4cc-59c5-4da3-b31d-222f182f5471] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-10T19:38:57.094 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: -2092> 2023-12-10T19:28:50.914+0000 7f2c61de2640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 62746) UID: 1000 2023-12-10T19:38:57.098 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: 0> 2023-12-10T19:38:57.062+0000 7f2c64e27880 -1 *** Caught signal (Aborted) ** 2023-12-10T19:38:57.098 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: in thread 7f2c64e27880 thread_name:rbd-mirror 2023-12-10T19:38:57.098 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: 2023-12-10T19:38:57.099 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: ceph version 19.0.0-130-g3d2db44f (3d2db44fa991bc355074aa9b7647f63b77201fef) reef (dev) 2023-12-10T19:38:57.099 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f2c65cba520] 2023-12-10T19:38:57.099 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: 2: /lib/x86_64-linux-gnu/libc.so.6(+0x91197) [0x7f2c65d09197] 2023-12-10T19:38:57.099 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: 3: pthread_cond_wait() 2023-12-10T19:38:57.099 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: 4: /lib/librados.so.2(+0x9be8f) [0x7f2c67324e8f] 2023-12-10T19:38:57.099 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: 5: (librados::v14_2_0::IoCtx::operate(std::__cxx11::basic_string, std::allocator > const&, librados::v14_2_0::ObjectReadOperation*, ceph::buffer::v15_2_0::list*)+0x65) [0x7f2c67308785] 2023-12-10T19:38:57.099 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: 6: (librbd::cls_client::mirror_mode_get(librados::v14_2_0::IoCtx*, cls::rbd::MirrorMode*)+0xa8) [0x5655530e0218] 2023-12-10T19:38:57.099 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: 7: (rbd::mirror::ClusterWatcher::read_pool_peers(std::map >, std::less, std::allocator > > > >*)+0x252) [0x565552bdf912] 2023-12-10T19:38:57.099 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: 8: (rbd::mirror::ClusterWatcher::refresh_pools()+0x78) [0x565552be1488] 2023-12-10T19:38:57.099 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: 9: (rbd::mirror::Mirror::run()+0xb5) [0x565552bb8935] 2023-12-10T19:38:57.099 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: 10: main() 2023-12-10T19:38:57.099 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: 11: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f2c65ca1d90] 2023-12-10T19:38:57.099 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: 12: __libc_start_main() 2023-12-10T19:38:57.099 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: 13: _start() 2023-12-10T19:38:57.100 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-10T19:38:57.100 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: 2023-12-10T19:38:57.102 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: -9999> 2023-12-10T19:20:22.076+0000 7f2c5add4640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5655570e2b60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-10T19:38:57.102 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: -9998> 2023-12-10T19:20:22.076+0000 7f2c60de0640 -1 rbd::mirror::ImageReplayer: 0x5655574bcc80 [3/195ff7a6-d9ef-4fb6-afac-e87b92563048] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-10T19:38:57.102 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: -9997> 2023-12-10T19:20:38.083+0000 7f2c5a5d3640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5655570e29c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-10T19:38:57.102 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: -9996> 2023-12-10T19:20:38.083+0000 7f2c615e1640 -1 rbd::mirror::ImageReplayer: 0x5655574bc780 [3/0caf4caf-9ee1-436f-874d-71291f8008fb] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-10T19:38:57.102 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: -9995> 2023-12-10T19:20:46.206+0000 7f2c5a5d3640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56555717e9c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-10T19:38:57.102 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: -9994> 2023-12-10T19:20:46.206+0000 7f2c60de0640 -1 rbd::mirror::ImageReplayer: 0x5655574bd680 [3/40a91520-4e7a-4ba5-aaf0-64312a2dcb5a] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-10T19:38:57.102 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: -9993> 2023-12-10T19:20:48.410+0000 7f2c5add4640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5655570e3520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-10T19:38:57.102 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: -9992> 2023-12-10T19:20:48.410+0000 7f2c615e1640 -1 rbd::mirror::ImageReplayer: 0x565556ff0500 [3/509368f6-035e-4be6-926a-dc4bce662e88] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-10T19:38:57.102 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: -9991> 2023-12-10T19:25:06.040+0000 7f2c5add4640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5655570d8340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-10T19:38:57.102 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: -9990> 2023-12-10T19:25:06.040+0000 7f2c60de0640 -1 rbd::mirror::ImageReplayer: 0x5655574bd400 [3/06e26537-406b-48fb-96a9-75ee5f7af95c] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-10T19:38:57.103 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: -9989> 2023-12-10T19:26:37.049+0000 7f2c5add4640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x565556fcb520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-10T19:38:57.103 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: -9988> 2023-12-10T19:26:37.049+0000 7f2c615e1640 -1 rbd::mirror::ImageReplayer: 0x565556ff0500 [3/674cd4cc-59c5-4da3-b31d-222f182f5471] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-10T19:38:57.103 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: -9987> 2023-12-10T19:28:50.914+0000 7f2c61de2640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 62746) UID: 1000 2023-12-10T19:38:57.103 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: -9986> 2023-12-10T19:38:57.062+0000 7f2c64e27880 -1 *** Caught signal (Aborted) ** 2023-12-10T19:38:57.103 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: in thread 7f2c64e27880 thread_name:rbd-mirror 2023-12-10T19:38:57.103 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: 2023-12-10T19:38:57.103 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: ceph version 19.0.0-130-g3d2db44f (3d2db44fa991bc355074aa9b7647f63b77201fef) reef (dev) 2023-12-10T19:38:57.103 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f2c65cba520] 2023-12-10T19:38:57.103 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: 2: /lib/x86_64-linux-gnu/libc.so.6(+0x91197) [0x7f2c65d09197] 2023-12-10T19:38:57.104 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: 3: pthread_cond_wait() 2023-12-10T19:38:57.104 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: 4: /lib/librados.so.2(+0x9be8f) [0x7f2c67324e8f] 2023-12-10T19:38:57.104 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: 5: (librados::v14_2_0::IoCtx::operate(std::__cxx11::basic_string, std::allocator > const&, librados::v14_2_0::ObjectReadOperation*, ceph::buffer::v15_2_0::list*)+0x65) [0x7f2c67308785] 2023-12-10T19:38:57.104 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: 6: (librbd::cls_client::mirror_mode_get(librados::v14_2_0::IoCtx*, cls::rbd::MirrorMode*)+0xa8) [0x5655530e0218] 2023-12-10T19:38:57.104 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: 7: (rbd::mirror::ClusterWatcher::read_pool_peers(std::map >, std::less, std::allocator > > > >*)+0x252) [0x565552bdf912] 2023-12-10T19:38:57.104 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: 8: (rbd::mirror::ClusterWatcher::refresh_pools()+0x78) [0x565552be1488] 2023-12-10T19:38:57.104 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: 9: (rbd::mirror::Mirror::run()+0xb5) [0x565552bb8935] 2023-12-10T19:38:57.104 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: 10: main() 2023-12-10T19:38:57.104 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: 11: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f2c65ca1d90] 2023-12-10T19:38:57.104 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: 12: __libc_start_main() 2023-12-10T19:38:57.104 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: 13: _start() 2023-12-10T19:38:57.104 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-10T19:38:57.104 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr: 2023-12-10T19:38:57.862 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi076.stderr:daemon-helper: command crashed with signal 6 2023-12-10T19:41:10.101 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:2023-12-10T19:41:10.098+0000 7fa9d8434640 0 monclient(hunting): authenticate timed out after 300 2023-12-10T19:41:10.102 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2023-12-10T19:41:10.105 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ CEPH_ARGS= ceph --cluster cluster2 service status 2023-12-10T19:41:27.046 INFO:tasks.rbd_mirror_thrash:Failed to stop cluster1.client.mirror.0 2023-12-10T19:41:27.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 6 2023-12-10T19:41:27.047 ERROR:tasks.rbd_mirror_thrash:exception: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/rbd_mirror_thrash.py", line 84, in _run self.do_thrash() File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/rbd_mirror_thrash.py", line 156, in do_thrash run.wait([daemon.proc], timeout=600) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (101) after waiting for 600 seconds 2023-12-10T19:41:27.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr:*** Caught signal (Aborted) ** 2023-12-10T19:41:27.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: in thread 7f1b83c35880 thread_name:rbd-mirror 2023-12-10T19:41:27.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: ceph version 19.0.0-130-g3d2db44f (3d2db44fa991bc355074aa9b7647f63b77201fef) reef (dev) 2023-12-10T19:41:27.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f1b84ac8520] 2023-12-10T19:41:27.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: 2: /lib/x86_64-linux-gnu/libc.so.6(+0x91197) [0x7f1b84b17197] 2023-12-10T19:41:27.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: 3: pthread_cond_wait() 2023-12-10T19:41:27.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: 4: rbd-mirror(+0x3725eb) [0x55f6dc3cd5eb] 2023-12-10T19:41:27.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: 5: (rbd::mirror::PoolReplayer::shut_down()+0x103) [0x55f6dc3d6e33] 2023-12-10T19:41:27.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: 6: (rbd::mirror::PoolReplayer::~PoolReplayer()+0x10) [0x55f6dc3d71d0] 2023-12-10T19:41:27.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: 7: rbd-mirror(+0x944178) [0x55f6dc99f178] 2023-12-10T19:41:27.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: 8: rbd-mirror(+0x944160) [0x55f6dc99f160] 2023-12-10T19:41:27.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: 9: (rbd::mirror::Mirror::~Mirror()+0xca) [0x55f6dc3c761a] 2023-12-10T19:41:27.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: 10: main() 2023-12-10T19:41:27.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: 11: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f1b84aafd90] 2023-12-10T19:41:27.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: 12: __libc_start_main() 2023-12-10T19:41:27.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: 13: _start() 2023-12-10T19:41:27.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr:2023-12-10T19:41:27.049+0000 7f1b83c35880 -1 *** Caught signal (Aborted) ** 2023-12-10T19:41:27.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: in thread 7f1b83c35880 thread_name:rbd-mirror 2023-12-10T19:41:27.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: 2023-12-10T19:41:27.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: ceph version 19.0.0-130-g3d2db44f (3d2db44fa991bc355074aa9b7647f63b77201fef) reef (dev) 2023-12-10T19:41:27.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f1b84ac8520] 2023-12-10T19:41:27.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: 2: /lib/x86_64-linux-gnu/libc.so.6(+0x91197) [0x7f1b84b17197] 2023-12-10T19:41:27.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: 3: pthread_cond_wait() 2023-12-10T19:41:27.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: 4: rbd-mirror(+0x3725eb) [0x55f6dc3cd5eb] 2023-12-10T19:41:27.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: 5: (rbd::mirror::PoolReplayer::shut_down()+0x103) [0x55f6dc3d6e33] 2023-12-10T19:41:27.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: 6: (rbd::mirror::PoolReplayer::~PoolReplayer()+0x10) [0x55f6dc3d71d0] 2023-12-10T19:41:27.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: 7: rbd-mirror(+0x944178) [0x55f6dc99f178] 2023-12-10T19:41:27.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: 8: rbd-mirror(+0x944160) [0x55f6dc99f160] 2023-12-10T19:41:27.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: 9: (rbd::mirror::Mirror::~Mirror()+0xca) [0x55f6dc3c761a] 2023-12-10T19:41:27.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: 10: main() 2023-12-10T19:41:27.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: 11: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f1b84aafd90] 2023-12-10T19:41:27.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: 12: __libc_start_main() 2023-12-10T19:41:27.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: 13: _start() 2023-12-10T19:41:27.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-10T19:41:27.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: 2023-12-10T19:41:27.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: -2746> 2023-12-10T19:31:22.915+0000 7f1b80bf0640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 72689) UID: 1000 2023-12-10T19:41:27.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: 0> 2023-12-10T19:41:27.049+0000 7f1b83c35880 -1 *** Caught signal (Aborted) ** 2023-12-10T19:41:27.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: in thread 7f1b83c35880 thread_name:rbd-mirror 2023-12-10T19:41:27.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: 2023-12-10T19:41:27.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: ceph version 19.0.0-130-g3d2db44f (3d2db44fa991bc355074aa9b7647f63b77201fef) reef (dev) 2023-12-10T19:41:27.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f1b84ac8520] 2023-12-10T19:41:27.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: 2: /lib/x86_64-linux-gnu/libc.so.6(+0x91197) [0x7f1b84b17197] 2023-12-10T19:41:27.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: 3: pthread_cond_wait() 2023-12-10T19:41:27.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: 4: rbd-mirror(+0x3725eb) [0x55f6dc3cd5eb] 2023-12-10T19:41:27.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: 5: (rbd::mirror::PoolReplayer::shut_down()+0x103) [0x55f6dc3d6e33] 2023-12-10T19:41:27.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: 6: (rbd::mirror::PoolReplayer::~PoolReplayer()+0x10) [0x55f6dc3d71d0] 2023-12-10T19:41:27.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: 7: rbd-mirror(+0x944178) [0x55f6dc99f178] 2023-12-10T19:41:27.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: 8: rbd-mirror(+0x944160) [0x55f6dc99f160] 2023-12-10T19:41:27.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: 9: (rbd::mirror::Mirror::~Mirror()+0xca) [0x55f6dc3c761a] 2023-12-10T19:41:27.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: 10: main() 2023-12-10T19:41:27.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: 11: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f1b84aafd90] 2023-12-10T19:41:27.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: 12: __libc_start_main() 2023-12-10T19:41:27.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: 13: _start() 2023-12-10T19:41:27.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-10T19:41:27.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: 2023-12-10T19:41:27.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: -8273> 2023-12-10T19:31:22.915+0000 7f1b80bf0640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 72689) UID: 1000 2023-12-10T19:41:27.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: -8272> 2023-12-10T19:41:27.049+0000 7f1b83c35880 -1 *** Caught signal (Aborted) ** 2023-12-10T19:41:27.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: in thread 7f1b83c35880 thread_name:rbd-mirror 2023-12-10T19:41:27.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: 2023-12-10T19:41:27.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: ceph version 19.0.0-130-g3d2db44f (3d2db44fa991bc355074aa9b7647f63b77201fef) reef (dev) 2023-12-10T19:41:27.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f1b84ac8520] 2023-12-10T19:41:27.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: 2: /lib/x86_64-linux-gnu/libc.so.6(+0x91197) [0x7f1b84b17197] 2023-12-10T19:41:27.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: 3: pthread_cond_wait() 2023-12-10T19:41:27.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: 4: rbd-mirror(+0x3725eb) [0x55f6dc3cd5eb] 2023-12-10T19:41:27.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: 5: (rbd::mirror::PoolReplayer::shut_down()+0x103) [0x55f6dc3d6e33] 2023-12-10T19:41:27.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: 6: (rbd::mirror::PoolReplayer::~PoolReplayer()+0x10) [0x55f6dc3d71d0] 2023-12-10T19:41:27.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: 7: rbd-mirror(+0x944178) [0x55f6dc99f178] 2023-12-10T19:41:27.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: 8: rbd-mirror(+0x944160) [0x55f6dc99f160] 2023-12-10T19:41:27.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: 9: (rbd::mirror::Mirror::~Mirror()+0xca) [0x55f6dc3c761a] 2023-12-10T19:41:27.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: 10: main() 2023-12-10T19:41:27.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: 11: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f1b84aafd90] 2023-12-10T19:41:27.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: 12: __libc_start_main() 2023-12-10T19:41:27.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: 13: _start() 2023-12-10T19:41:27.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-10T19:41:27.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr: 2023-12-10T19:41:27.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi076.stderr:daemon-helper: command crashed with signal 6 2023-12-10T19:41:27.712 INFO:tasks.daemonwatchdog.daemon_watchdog:thrasher.rbd_mirror.[cluster1] failed 2023-12-10T19:41:27.712 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2023-12-10T19:41:28.218 INFO:tasks.ceph.cluster1.osd.0:Sent signal 15 2023-12-10T19:41:28.218 INFO:tasks.ceph.cluster1.osd.1:Sent signal 15 2023-12-10T19:41:28.218 INFO:tasks.ceph.cluster1.osd.2:Sent signal 15 2023-12-10T19:41:28.218 INFO:tasks.ceph.cluster1.mon.a:Sent signal 15 2023-12-10T19:41:28.219 INFO:tasks.ceph.cluster1.mgr.x:Sent signal 15 2023-12-10T19:41:28.219 INFO:tasks.ceph.cluster1.osd.0.smithi033.stderr:2023-12-10T19:41:28.212+0000 7f837d6e2640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster cluster1 -i 0 (PID: 34760) UID: 0 2023-12-10T19:41:28.219 INFO:tasks.ceph.cluster1.osd.0.smithi033.stderr:2023-12-10T19:41:28.212+0000 7f837d6e2640 -1 osd.0 459 *** Got signal Terminated *** 2023-12-10T19:41:28.219 INFO:tasks.ceph.cluster1.osd.0.smithi033.stderr:2023-12-10T19:41:28.212+0000 7f837d6e2640 -1 osd.0 459 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-10T19:41:28.220 INFO:tasks.ceph.cluster1.osd.1.smithi033.stderr:2023-12-10T19:41:28.212+0000 7f06b32b9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster cluster1 -i 1 (PID: 34763) UID: 0 2023-12-10T19:41:28.220 INFO:tasks.ceph.cluster1.osd.1.smithi033.stderr:2023-12-10T19:41:28.212+0000 7f06b32b9640 -1 osd.1 459 *** Got signal Terminated *** 2023-12-10T19:41:28.220 INFO:tasks.ceph.cluster1.osd.1.smithi033.stderr:2023-12-10T19:41:28.212+0000 7f06b32b9640 -1 osd.1 459 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-10T19:41:28.220 INFO:tasks.ceph.cluster1.osd.2.smithi033.stderr:2023-12-10T19:41:28.216+0000 7fef9892f640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster cluster1 -i 2 (PID: 34766) UID: 0 2023-12-10T19:41:28.220 INFO:tasks.ceph.cluster1.osd.2.smithi033.stderr:2023-12-10T19:41:28.216+0000 7fef9892f640 -1 osd.2 459 *** Got signal Terminated *** 2023-12-10T19:41:28.220 INFO:tasks.ceph.cluster1.osd.2.smithi033.stderr:2023-12-10T19:41:28.216+0000 7fef9892f640 -1 osd.2 459 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-10T19:41:28.221 INFO:tasks.ceph.cluster1.mon.a.smithi033.stderr:2023-12-10T19:41:28.216+0000 7fc5771ba640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mon -f --cluster cluster1 -i a (PID: 34588) UID: 0 2023-12-10T19:41:28.221 INFO:tasks.ceph.cluster1.mon.a.smithi033.stderr:2023-12-10T19:41:28.216+0000 7fc5771ba640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-12-10T19:41:28.221 INFO:tasks.ceph.cluster1.mgr.x.smithi033.stderr:2023-12-10T19:41:28.216+0000 7f6b08e77640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mgr -f --cluster cluster1 -i x (PID: 34590) UID: 0 2023-12-10T19:41:28.221 INFO:tasks.ceph.cluster1.mgr.x.smithi033.stderr:2023-12-10T19:41:28.216+0000 7f6b08e77640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2023-12-10T19:46:10.224 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:2023-12-10T19:46:10.220+0000 7fdd9c286640 0 monclient(hunting): authenticate timed out after 300 2023-12-10T19:46:10.224 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2023-12-10T19:46:10.228 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:46:10.229 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster2 mirror images 2023-12-10T19:46:10.229 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo 2023-12-10T19:46:10.229 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2 mirror images 2023-12-10T19:46:10.229 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror --namespace ls -l 2023-12-10T19:51:10.264 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: couldn't connect to the cluster! 2023-12-10T19:51:10.265 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: listing images failed: (110) Connection timed out 2023-12-10T19:51:10.266 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:51:10.266 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster2 mirror mirror pool info 2023-12-10T19:51:10.266 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo 2023-12-10T19:51:10.266 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2 mirror mirror pool info 2023-12-10T19:51:10.266 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror --namespace mirror pool info 2023-12-10T19:56:10.304 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: couldn't connect to the cluster! 2023-12-10T19:56:10.307 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo 2023-12-10T19:56:10.307 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T19:56:10.307 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster2 mirror mirror pool status 2023-12-10T19:56:10.307 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2 mirror mirror pool status 2023-12-10T19:56:10.307 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ CEPH_ARGS= rbd --cluster cluster2 -p mirror --namespace mirror pool status --verbose 2023-12-10T20:01:10.344 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: couldn't connect to the cluster! 2023-12-10T20:01:10.346 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T20:01:10.346 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo 2023-12-10T20:01:10.347 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror --namespace ls 2023-12-10T20:06:10.386 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster2 mirror rbd_mirroring omap vals 2023-12-10T20:06:10.386 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2 mirror rbd_mirroring omap vals 2023-12-10T20:06:10.387 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rados --cluster cluster2 -p mirror --namespace listomapvals rbd_mirroring 2023-12-10T20:11:10.420 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-10T20:11:10.421 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2 mirror rbd_mirror_leader omap vals 2023-12-10T20:11:10.422 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster2 mirror rbd_mirror_leader omap vals 2023-12-10T20:11:10.422 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rados --cluster cluster2 -p mirror --namespace listomapvals rbd_mirror_leader 2023-12-10T20:16:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-10T20:16:10.456 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T20:16:10.456 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo 2023-12-10T20:16:10.456 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2 mirror ns1 images 2023-12-10T20:16:10.456 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster2 mirror ns1 images 2023-12-10T20:16:10.457 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror --namespace ns1 ls -l 2023-12-10T20:21:10.490 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: couldn't connect to the cluster! 2023-12-10T20:21:10.490 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: listing images failed: (110) Connection timed out 2023-12-10T20:21:10.493 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T20:21:10.493 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster2 mirrorns1 mirror pool info 2023-12-10T20:21:10.493 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo 2023-12-10T20:21:10.493 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2 mirrorns1 mirror pool info 2023-12-10T20:21:10.493 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror --namespace ns1 mirror pool info 2023-12-10T20:26:10.531 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: couldn't connect to the cluster! 2023-12-10T20:26:10.559 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T20:26:10.559 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster2 mirrorns1 mirror pool status 2023-12-10T20:26:10.559 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo 2023-12-10T20:26:10.559 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2 mirrorns1 mirror pool status 2023-12-10T20:26:10.559 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ CEPH_ARGS= rbd --cluster cluster2 -p mirror --namespace ns1 mirror pool status --verbose 2023-12-10T20:31:10.570 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: couldn't connect to the cluster! 2023-12-10T20:31:10.572 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T20:31:10.573 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo 2023-12-10T20:31:10.573 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror --namespace ns1 ls 2023-12-10T20:36:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster2 mirror ns1 rbd_mirroring omap vals 2023-12-10T20:36:10.614 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2 mirror ns1 rbd_mirroring omap vals 2023-12-10T20:36:10.614 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rados --cluster cluster2 -p mirror --namespace ns1 listomapvals rbd_mirroring 2023-12-10T20:41:10.645 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-10T20:41:10.646 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2 mirror ns1 rbd_mirror_leader omap vals 2023-12-10T20:41:10.647 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster2 mirror ns1 rbd_mirror_leader omap vals 2023-12-10T20:41:10.647 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rados --cluster cluster2 -p mirror --namespace ns1 listomapvals rbd_mirror_leader 2023-12-10T20:46:10.680 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-10T20:46:10.681 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T20:46:10.681 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster2 mirror ns2 images 2023-12-10T20:46:10.681 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo 2023-12-10T20:46:10.681 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2 mirror ns2 images 2023-12-10T20:46:10.681 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror --namespace ns2 ls -l 2023-12-10T20:51:10.717 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: couldn't connect to the cluster! 2023-12-10T20:51:10.717 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: listing images failed: (110) Connection timed out 2023-12-10T20:51:10.719 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T20:51:10.719 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster2 mirrorns2 mirror pool info 2023-12-10T20:51:10.720 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo 2023-12-10T20:51:10.720 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2 mirrorns2 mirror pool info 2023-12-10T20:51:10.720 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror --namespace ns2 mirror pool info 2023-12-10T20:56:10.757 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: couldn't connect to the cluster! 2023-12-10T20:56:10.760 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T20:56:10.760 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster2 mirrorns2 mirror pool status 2023-12-10T20:56:10.760 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo 2023-12-10T20:56:10.760 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2 mirrorns2 mirror pool status 2023-12-10T20:56:10.760 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ CEPH_ARGS= rbd --cluster cluster2 -p mirror --namespace ns2 mirror pool status --verbose 2023-12-10T21:01:10.799 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: couldn't connect to the cluster! 2023-12-10T21:01:10.800 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T21:01:10.800 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo 2023-12-10T21:01:10.800 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror --namespace ns2 ls 2023-12-10T21:06:10.839 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster2 mirror ns2 rbd_mirroring omap vals 2023-12-10T21:06:10.839 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2 mirror ns2 rbd_mirroring omap vals 2023-12-10T21:06:10.840 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rados --cluster cluster2 -p mirror --namespace ns2 listomapvals rbd_mirroring 2023-12-10T21:11:10.872 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-10T21:11:10.873 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster2 mirror ns2 rbd_mirror_leader omap vals 2023-12-10T21:11:10.874 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2 mirror ns2 rbd_mirror_leader omap vals 2023-12-10T21:11:10.874 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rados --cluster cluster2 -p mirror --namespace ns2 listomapvals rbd_mirror_leader 2023-12-10T21:16:10.907 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-10T21:16:10.908 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T21:16:10.909 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster2 mirror_parent images 2023-12-10T21:16:10.909 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo 2023-12-10T21:16:10.909 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2 mirror_parent images 2023-12-10T21:16:10.909 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace ls -l 2023-12-10T21:21:10.947 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: couldn't connect to the cluster! 2023-12-10T21:21:10.947 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: listing images failed: (110) Connection timed out 2023-12-10T21:21:10.949 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T21:21:10.949 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster2 mirror_parent mirror pool info 2023-12-10T21:21:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo 2023-12-10T21:21:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2 mirror_parent mirror pool info 2023-12-10T21:21:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace mirror pool info 2023-12-10T21:26:10.987 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: couldn't connect to the cluster! 2023-12-10T21:26:10.989 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo 2023-12-10T21:26:10.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T21:26:10.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster2 mirror_parent mirror pool status 2023-12-10T21:26:10.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2 mirror_parent mirror pool status 2023-12-10T21:26:10.990 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ CEPH_ARGS= rbd --cluster cluster2 -p mirror_parent --namespace mirror pool status --verbose 2023-12-10T21:31:11.029 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: couldn't connect to the cluster! 2023-12-10T21:31:11.031 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T21:31:11.032 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo 2023-12-10T21:31:11.032 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace ls 2023-12-10T21:36:11.071 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster2 mirror_parent rbd_mirroring omap vals 2023-12-10T21:36:11.072 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2 mirror_parent rbd_mirroring omap vals 2023-12-10T21:36:11.072 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rados --cluster cluster2 -p mirror_parent --namespace listomapvals rbd_mirroring 2023-12-10T21:41:11.104 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-10T21:41:11.105 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster2 mirror_parent rbd_mirror_leader omap vals 2023-12-10T21:41:11.105 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2 mirror_parent rbd_mirror_leader omap vals 2023-12-10T21:41:11.106 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rados --cluster cluster2 -p mirror_parent --namespace listomapvals rbd_mirror_leader 2023-12-10T21:46:11.139 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-10T21:46:11.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo 2023-12-10T21:46:11.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T21:46:11.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster2 mirror_parent ns1 images 2023-12-10T21:46:11.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2 mirror_parent ns1 images 2023-12-10T21:46:11.140 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace ns1 ls -l 2023-12-10T21:51:11.175 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: couldn't connect to the cluster! 2023-12-10T21:51:11.176 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: listing images failed: (110) Connection timed out 2023-12-10T21:51:11.178 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T21:51:11.178 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster2 mirror_parentns1 mirror pool info 2023-12-10T21:51:11.178 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo 2023-12-10T21:51:11.178 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2 mirror_parentns1 mirror pool info 2023-12-10T21:51:11.178 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace ns1 mirror pool info 2023-12-10T21:56:11.216 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: couldn't connect to the cluster! 2023-12-10T21:56:11.219 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T21:56:11.219 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster2 mirror_parentns1 mirror pool status 2023-12-10T21:56:11.219 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo 2023-12-10T21:56:11.219 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2 mirror_parentns1 mirror pool status 2023-12-10T21:56:11.219 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ CEPH_ARGS= rbd --cluster cluster2 -p mirror_parent --namespace ns1 mirror pool status --verbose 2023-12-10T22:01:11.258 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: couldn't connect to the cluster! 2023-12-10T22:01:11.261 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T22:01:11.261 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo 2023-12-10T22:01:11.262 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace ns1 ls 2023-12-10T22:06:11.299 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster2 mirror_parent ns1 rbd_mirroring omap vals 2023-12-10T22:06:11.300 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2 mirror_parent ns1 rbd_mirroring omap vals 2023-12-10T22:06:11.300 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rados --cluster cluster2 -p mirror_parent --namespace ns1 listomapvals rbd_mirroring 2023-12-10T22:11:11.332 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-10T22:11:11.333 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster2 mirror_parent ns1 rbd_mirror_leader omap vals 2023-12-10T22:11:11.333 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2 mirror_parent ns1 rbd_mirror_leader omap vals 2023-12-10T22:11:11.333 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rados --cluster cluster2 -p mirror_parent --namespace ns1 listomapvals rbd_mirror_leader 2023-12-10T22:16:11.367 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-10T22:16:11.368 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo 2023-12-10T22:16:11.369 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T22:16:11.369 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster2 mirror_parent ns2 images 2023-12-10T22:16:11.369 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2 mirror_parent ns2 images 2023-12-10T22:16:11.369 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace ns2 ls -l 2023-12-10T22:21:11.408 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: couldn't connect to the cluster! 2023-12-10T22:21:11.410 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: listing images failed: (110) Connection timed out 2023-12-10T22:21:11.410 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T22:21:11.410 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster2 mirror_parentns2 mirror pool info 2023-12-10T22:21:11.410 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo 2023-12-10T22:21:11.410 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2 mirror_parentns2 mirror pool info 2023-12-10T22:21:11.410 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace ns2 mirror pool info 2023-12-10T22:26:11.446 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: couldn't connect to the cluster! 2023-12-10T22:26:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T22:26:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster2 mirror_parentns2 mirror pool status 2023-12-10T22:26:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo 2023-12-10T22:26:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2 mirror_parentns2 mirror pool status 2023-12-10T22:26:11.450 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ CEPH_ARGS= rbd --cluster cluster2 -p mirror_parent --namespace ns2 mirror pool status --verbose 2023-12-10T22:31:11.485 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:rbd: couldn't connect to the cluster! 2023-12-10T22:31:11.487 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo 2023-12-10T22:31:11.488 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T22:31:11.488 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace ns2 ls 2023-12-10T22:36:11.527 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster2 mirror_parent ns2 rbd_mirroring omap vals 2023-12-10T22:36:11.527 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2 mirror_parent ns2 rbd_mirroring omap vals 2023-12-10T22:36:11.527 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rados --cluster cluster2 -p mirror_parent --namespace ns2 listomapvals rbd_mirroring 2023-12-10T22:41:11.559 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-10T22:41:11.560 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster2 mirror_parent ns2 rbd_mirror_leader omap vals 2023-12-10T22:41:11.560 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2 mirror_parent ns2 rbd_mirror_leader omap vals 2023-12-10T22:41:11.560 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rados --cluster cluster2 -p mirror_parent --namespace ns2 listomapvals rbd_mirror_leader 2023-12-10T22:46:11.595 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-10T22:46:11.596 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo 2023-12-10T22:46:11.597 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout: 2023-12-10T22:46:11.597 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local ret 2023-12-10T22:46:11.597 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T22:46:11.599 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_pid_file cluster1:0 2023-12-10T22:46:11.599 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T22:46:11.599 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T22:46:11.599 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-10T22:46:11.599 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:0 2023-12-10T22:46:11.599 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T22:46:11.600 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T22:46:11.600 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T22:46:11.600 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T22:46:11.600 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster1:0 ] 2023-12-10T22:46:11.600 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T22:46:11.600 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T22:46:11.600 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T22:46:11.600 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T22:46:11.600 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 pid file 2023-12-10T22:46:11.624 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2023-12-10T22:46:11.625 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pid_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2023-12-10T22:46:11.625 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ ! -e /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid ] 2023-12-10T22:46:11.625 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1 rbd-mirror not running or unknown (/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid not exist) 2023-12-10T22:46:11.625 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster1 rbd-mirror not running or unknown (/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid not exist) 2023-12-10T22:46:11.626 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ continue 2023-12-10T22:46:11.627 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_pid_file cluster1:1 2023-12-10T22:46:11.627 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:1 2023-12-10T22:46:11.627 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T22:46:11.627 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-10T22:46:11.627 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:1 2023-12-10T22:46:11.627 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T22:46:11.627 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T22:46:11.627 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T22:46:11.627 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T22:46:11.627 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster1:1 ] 2023-12-10T22:46:11.628 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T22:46:11.628 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T22:46:11.628 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T22:46:11.628 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T22:46:11.628 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 pid file 2023-12-10T22:46:11.650 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2023-12-10T22:46:11.650 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster1 rbd-mirror not running or unknown (/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid not exist) 2023-12-10T22:46:11.651 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pid_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2023-12-10T22:46:11.651 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ ! -e /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid ] 2023-12-10T22:46:11.651 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1 rbd-mirror not running or unknown (/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid not exist) 2023-12-10T22:46:11.652 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ continue 2023-12-10T22:46:11.652 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_pid_file cluster1:2 2023-12-10T22:46:11.652 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:2 2023-12-10T22:46:11.652 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T22:46:11.652 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-10T22:46:11.652 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:2 2023-12-10T22:46:11.652 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T22:46:11.652 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T22:46:11.652 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T22:46:11.652 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T22:46:11.653 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster1:2 ] 2023-12-10T22:46:11.653 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T22:46:11.653 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T22:46:11.653 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T22:46:11.653 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T22:46:11.653 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 pid file 2023-12-10T22:46:11.671 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2023-12-10T22:46:11.672 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pid_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2023-12-10T22:46:11.672 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster1 rbd-mirror not running or unknown (/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid not exist) 2023-12-10T22:46:11.673 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ ! -e /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid ] 2023-12-10T22:46:11.673 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1 rbd-mirror not running or unknown (/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid not exist) 2023-12-10T22:46:11.673 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ continue 2023-12-10T22:46:11.673 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_pid_file cluster1:3 2023-12-10T22:46:11.673 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:3 2023-12-10T22:46:11.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T22:46:11.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-10T22:46:11.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster1:3 2023-12-10T22:46:11.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T22:46:11.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T22:46:11.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T22:46:11.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T22:46:11.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 3 = cluster1:3 ] 2023-12-10T22:46:11.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster1 2023-12-10T22:46:11.674 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster1 2023-12-10T22:46:11.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=3 2023-12-10T22:46:11.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T22:46:11.675 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 pid file 2023-12-10T22:46:11.693 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2023-12-10T22:46:11.693 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pid_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2023-12-10T22:46:11.693 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster1 rbd-mirror not running or unknown (/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid not exist) 2023-12-10T22:46:11.696 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ ! -e /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid ] 2023-12-10T22:46:11.696 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster1 rbd-mirror not running or unknown (/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid not exist) 2023-12-10T22:46:11.696 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ continue 2023-12-10T22:46:11.696 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T22:46:11.696 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_pid_file cluster2:0 2023-12-10T22:46:11.696 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2:0 2023-12-10T22:46:11.696 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T22:46:11.696 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster2:0 cluster instance 2023-12-10T22:46:11.696 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster2:0 2023-12-10T22:46:11.696 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T22:46:11.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T22:46:11.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster2 2023-12-10T22:46:11.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T22:46:11.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 0 = cluster2:0 ] 2023-12-10T22:46:11.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster2 2023-12-10T22:46:11.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster2 2023-12-10T22:46:11.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=0 2023-12-10T22:46:11.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=0 2023-12-10T22:46:11.697 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.0 pid file 2023-12-10T22:46:11.714 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2023-12-10T22:46:11.715 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pid_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2023-12-10T22:46:11.715 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ ! -e /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid ] 2023-12-10T22:46:11.715 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster2 rbd-mirror not running or unknown (/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid not exist) 2023-12-10T22:46:11.716 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2 rbd-mirror not running or unknown (/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid not exist) 2023-12-10T22:46:11.717 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ continue 2023-12-10T22:46:11.717 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_pid_file cluster2:1 2023-12-10T22:46:11.717 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2:1 2023-12-10T22:46:11.717 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T22:46:11.717 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster2:1 cluster instance 2023-12-10T22:46:11.717 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster2:1 2023-12-10T22:46:11.717 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T22:46:11.717 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T22:46:11.717 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster2 2023-12-10T22:46:11.718 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T22:46:11.718 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 = cluster2:1 ] 2023-12-10T22:46:11.718 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster2 2023-12-10T22:46:11.718 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster2 2023-12-10T22:46:11.718 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=1 2023-12-10T22:46:11.718 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=1 2023-12-10T22:46:11.718 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.1 pid file 2023-12-10T22:46:11.735 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2023-12-10T22:46:11.735 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pid_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2023-12-10T22:46:11.735 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ ! -e /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid ] 2023-12-10T22:46:11.735 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster2 rbd-mirror not running or unknown (/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid not exist) 2023-12-10T22:46:11.737 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2 rbd-mirror not running or unknown (/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid not exist) 2023-12-10T22:46:11.737 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ continue 2023-12-10T22:46:11.737 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_pid_file cluster2:2 2023-12-10T22:46:11.737 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2:2 2023-12-10T22:46:11.737 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T22:46:11.737 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster2:2 cluster instance 2023-12-10T22:46:11.737 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster2:2 2023-12-10T22:46:11.737 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T22:46:11.737 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T22:46:11.738 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster2 2023-12-10T22:46:11.738 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T22:46:11.738 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 2 = cluster2:2 ] 2023-12-10T22:46:11.738 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster2 2023-12-10T22:46:11.738 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster2 2023-12-10T22:46:11.738 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=2 2023-12-10T22:46:11.738 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=2 2023-12-10T22:46:11.738 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.2 pid file 2023-12-10T22:46:11.755 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.pid 2023-12-10T22:46:11.755 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pid_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.pid 2023-12-10T22:46:11.755 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ ! -e /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.pid ] 2023-12-10T22:46:11.755 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster2 rbd-mirror not running or unknown (/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.pid not exist) 2023-12-10T22:46:11.757 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2 rbd-mirror not running or unknown (/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.pid not exist) 2023-12-10T22:46:11.757 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ continue 2023-12-10T22:46:11.757 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ daemon_pid_file cluster2:3 2023-12-10T22:46:11.757 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2:3 2023-12-10T22:46:11.757 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance 2023-12-10T22:46:11.757 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ set_cluster_instance cluster2:3 cluster instance 2023-12-10T22:46:11.758 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local val=cluster2:3 2023-12-10T22:46:11.758 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster_var_name=cluster 2023-12-10T22:46:11.758 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local instance_var_name=instance 2023-12-10T22:46:11.758 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster2 2023-12-10T22:46:11.758 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T22:46:11.758 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 3 = cluster2:3 ] 2023-12-10T22:46:11.758 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval cluster=cluster2 2023-12-10T22:46:11.758 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ cluster=cluster2 2023-12-10T22:46:11.758 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ eval instance=3 2023-12-10T22:46:11.758 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ instance=3 2023-12-10T22:46:11.758 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.3 pid file 2023-12-10T22:46:11.776 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.pid 2023-12-10T22:46:11.777 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local pid_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.pid 2023-12-10T22:46:11.777 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ ! -e /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.pid ] 2023-12-10T22:46:11.777 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:cluster2 rbd-mirror not running or unknown (/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.pid not exist) 2023-12-10T22:46:11.777 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo cluster2 rbd-mirror not running or unknown (/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.pid not exist) 2023-12-10T22:46:11.778 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ continue 2023-12-10T22:46:11.778 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 2023-12-10T22:46:11.778 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -z ] 2023-12-10T22:46:11.778 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster instance 2023-12-10T22:46:11.778 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ CEPH_ARGS= ceph --cluster cluster1 osd pool rm mirror mirror --yes-i-really-really-mean-it 2023-12-10T22:51:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:2023-12-10T22:51:11.889+0000 7f562f2ec640 0 monclient(hunting): authenticate timed out after 300 2023-12-10T22:51:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2023-12-10T22:51:11.894 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ CEPH_ARGS= ceph --cluster cluster2 osd pool rm mirror mirror --yes-i-really-really-mean-it 2023-12-10T22:56:12.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:2023-12-10T22:56:12.019+0000 7f4e6c26a640 0 monclient(hunting): authenticate timed out after 300 2023-12-10T22:56:12.022 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2023-12-10T22:56:12.025 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ CEPH_ARGS= ceph --cluster cluster1 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2023-12-10T23:01:12.145 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:2023-12-10T23:01:12.141+0000 7f80fda9f640 0 monclient(hunting): authenticate timed out after 300 2023-12-10T23:01:12.146 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2023-12-10T23:01:12.147 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ CEPH_ARGS= ceph --cluster cluster2 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2023-12-10T23:06:12.271 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:2023-12-10T23:06:12.267+0000 7f2085ad7640 0 monclient(hunting): authenticate timed out after 300 2023-12-10T23:06:12.272 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2023-12-10T23:06:12.275 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stop_mirrors cluster1 2023-12-10T23:06:12.275 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1 2023-12-10T23:06:12.275 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local sig= 2023-12-10T23:06:12.275 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T23:06:12.275 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stop_mirror cluster1:0 2023-12-10T23:06:12.275 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:0 2023-12-10T23:06:12.275 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local sig= 2023-12-10T23:06:12.275 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 1 2023-12-10T23:06:12.276 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 2023-12-10T23:06:12.276 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stop_mirror cluster1:1 2023-12-10T23:06:12.276 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:1 2023-12-10T23:06:12.276 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local sig= 2023-12-10T23:06:12.276 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 1 2023-12-10T23:06:12.276 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 2023-12-10T23:06:12.276 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stop_mirror cluster1:2 2023-12-10T23:06:12.276 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:2 2023-12-10T23:06:12.276 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local sig= 2023-12-10T23:06:12.276 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 1 2023-12-10T23:06:12.276 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 2023-12-10T23:06:12.276 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stop_mirror cluster1:3 2023-12-10T23:06:12.276 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster1:3 2023-12-10T23:06:12.276 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local sig= 2023-12-10T23:06:12.276 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 1 2023-12-10T23:06:12.276 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 2023-12-10T23:06:12.277 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stop_mirrors cluster2 2023-12-10T23:06:12.277 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2 2023-12-10T23:06:12.277 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local sig= 2023-12-10T23:06:12.277 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ seq 0 3 2023-12-10T23:06:12.277 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stop_mirror cluster2:0 2023-12-10T23:06:12.277 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2:0 2023-12-10T23:06:12.277 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local sig= 2023-12-10T23:06:12.277 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 1 2023-12-10T23:06:12.277 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 2023-12-10T23:06:12.277 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stop_mirror cluster2:1 2023-12-10T23:06:12.277 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2:1 2023-12-10T23:06:12.277 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local sig= 2023-12-10T23:06:12.277 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 1 2023-12-10T23:06:12.277 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 2023-12-10T23:06:12.278 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stop_mirror cluster2:2 2023-12-10T23:06:12.278 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2:2 2023-12-10T23:06:12.278 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local sig= 2023-12-10T23:06:12.278 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 1 2023-12-10T23:06:12.278 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 2023-12-10T23:06:12.278 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ stop_mirror cluster2:3 2023-12-10T23:06:12.278 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local cluster=cluster2:3 2023-12-10T23:06:12.279 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ local sig= 2023-12-10T23:06:12.279 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test -n 1 2023-12-10T23:06:12.279 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ return 2023-12-10T23:06:12.279 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ -z 1 ] 2023-12-10T23:06:12.279 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ test = /tmp/tmp.qNjRyMq75D 2023-12-10T23:06:12.279 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ rm -Rf /tmp/tmp.qNjRyMq75D 2023-12-10T23:06:12.279 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ [ 1 -eq 0 ] 2023-12-10T23:06:12.279 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stdout:FAIL 2023-12-10T23:06:12.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-10T23:06:12.281 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ echo FAIL 2023-12-10T23:06:12.281 INFO:tasks.workunit.cluster1.client.mirror.smithi076.stderr:+ exit 1 2023-12-10T23:06:12.281 INFO:tasks.workunit:Stopping ['rbd/rbd_mirror_stress.sh'] on cluster1.client.mirror... 2023-12-10T23:06:12.281 DEBUG:teuthology.orchestra.run.smithi076:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.cluster1.client.mirror /home/ubuntu/cephtest/clone.cluster1.client.mirror 2023-12-10T23:06:12.581 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/rbd_mirror_stress.sh) on smithi076 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3d2db44fa991bc355074aa9b7647f63b77201fef TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering RBD_MIRROR_INSTANCES=4 RBD_MIRROR_MODE=snapshot RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh' 2023-12-10T23:06:12.653 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=e6bc631a89974d6b8856b0f0c5b2243f Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/rbd_mirror_stress.sh) on smithi076 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3d2db44fa991bc355074aa9b7647f63b77201fef TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering RBD_MIRROR_INSTANCES=4 RBD_MIRROR_MODE=snapshot RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh' 2023-12-10T23:06:12.655 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror-thrash 2023-12-10T23:06:12.666 INFO:tasks.rbd_mirror_thrash:joining rbd_mirror_thrash 2023-12-10T23:06:12.666 ERROR:teuthology.run_tasks:Manager failed: rbd-mirror-thrash Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing 2023-12-10T23:06:12.672 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror-thrash 2023-12-10T23:06:12.683 INFO:tasks.rbd_mirror_thrash:joining rbd_mirror_thrash 2023-12-10T23:06:12.683 ERROR:teuthology.run_tasks:Manager failed: rbd-mirror-thrash Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/rbd_mirror_thrash.py", line 211, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing 2023-12-10T23:06:12.688 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-12-10T23:06:12.697 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.3:waiting for process to exit 2023-12-10T23:06:12.697 INFO:teuthology.orchestra.run:waiting for 300 2023-12-10T23:06:12.708 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi076.stderr:2023-12-10T23:06:12.699+0000 7f3eecc2c640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 73010) UID: 1000 2023-12-10T23:11:11.763 ERROR:teuthology.run_tasks:Manager failed: rbd-mirror Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/rbd_mirror.py", line 117, in end mirror_daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2023-12-10T23:11:11.773 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-12-10T23:11:11.785 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.2:waiting for process to exit 2023-12-10T23:11:11.786 INFO:teuthology.orchestra.run:waiting for 300 2023-12-10T23:11:11.790 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi076.stderr:2023-12-10T23:11:11.785+0000 7f354f0c9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 76184) UID: 1000 2023-12-10T23:16:10.860 ERROR:teuthology.run_tasks:Manager failed: rbd-mirror Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/rbd_mirror.py", line 117, in end mirror_daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2023-12-10T23:16:10.866 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-12-10T23:16:10.878 ERROR:tasks.rbd_mirror.cluster2.client.mirror.1:tried to stop a non-running daemon 2023-12-10T23:16:10.881 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-12-10T23:16:10.894 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.0:waiting for process to exit 2023-12-10T23:16:10.894 INFO:teuthology.orchestra.run:waiting for 300 2023-12-10T23:16:10.907 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi076.stderr:2023-12-10T23:16:10.895+0000 7f77474ee640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 75469) UID: 1000 2023-12-10T23:21:09.964 ERROR:teuthology.run_tasks:Manager failed: rbd-mirror Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/rbd_mirror.py", line 117, in end mirror_daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2023-12-10T23:21:09.970 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-12-10T23:21:09.981 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.3:waiting for process to exit 2023-12-10T23:21:09.981 INFO:teuthology.orchestra.run:waiting for 300 2023-12-10T23:21:09.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi076.stderr:2023-12-10T23:21:09.981+0000 7fe72dfee640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 74003) UID: 1000 2023-12-10T23:26:09.046 ERROR:teuthology.run_tasks:Manager failed: rbd-mirror Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/rbd_mirror.py", line 117, in end mirror_daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2023-12-10T23:26:09.055 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-12-10T23:26:09.070 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.2:waiting for process to exit 2023-12-10T23:26:09.070 INFO:teuthology.orchestra.run:waiting for 300 2023-12-10T23:26:09.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi076.stderr:2023-12-10T23:26:09.071+0000 7f3f0e221640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 59545) UID: 1000 2023-12-10T23:31:08.145 ERROR:teuthology.run_tasks:Manager failed: rbd-mirror Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/rbd_mirror.py", line 117, in end mirror_daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2023-12-10T23:31:08.162 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-12-10T23:31:08.178 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.1:waiting for process to exit 2023-12-10T23:31:08.179 INFO:teuthology.orchestra.run:waiting for 300 2023-12-10T23:31:08.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi076.stderr:2023-12-10T23:31:08.178+0000 7f1f30d6e640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 77188) UID: 1000 2023-12-10T23:36:07.250 ERROR:teuthology.run_tasks:Manager failed: rbd-mirror Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/rbd_mirror.py", line 117, in end mirror_daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2023-12-10T23:36:07.257 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-12-10T23:36:07.268 ERROR:tasks.rbd_mirror.cluster1.client.mirror.0:tried to stop a non-running daemon 2023-12-10T23:36:07.272 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-12-10T23:36:07.291 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2023-12-10T23:36:07.291 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-10T23:38:07.325 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-10T23:38:07.326 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi076 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' 2023-12-10T23:38:07.340 INFO:teuthology.misc:Shutting down mds daemons... 2023-12-10T23:38:07.341 INFO:teuthology.misc:Shutting down osd daemons... 2023-12-10T23:38:07.341 DEBUG:tasks.ceph.cluster2.osd.0:waiting for process to exit 2023-12-10T23:38:07.341 INFO:teuthology.orchestra.run:waiting for 300 2023-12-10T23:38:07.344 DEBUG:teuthology.orchestra.run:got remote process result: 5 2023-12-10T23:38:07.344 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi076 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi076 with status 5: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0' 2023-12-10T23:38:07.355 INFO:tasks.ceph.cluster2.osd.0:Stopped 2023-12-10T23:38:07.356 DEBUG:tasks.ceph.cluster2.osd.1:waiting for process to exit 2023-12-10T23:38:07.356 INFO:teuthology.orchestra.run:waiting for 300 2023-12-10T23:38:07.356 DEBUG:teuthology.orchestra.run:got remote process result: 5 2023-12-10T23:38:07.356 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi076 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi076 with status 5: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 1' 2023-12-10T23:38:07.368 INFO:tasks.ceph.cluster2.osd.1:Stopped 2023-12-10T23:38:07.368 DEBUG:tasks.ceph.cluster2.osd.2:waiting for process to exit 2023-12-10T23:38:07.368 INFO:teuthology.orchestra.run:waiting for 300 2023-12-10T23:38:07.368 INFO:tasks.ceph.cluster2.osd.2:Stopped 2023-12-10T23:38:07.368 INFO:teuthology.misc:Shutting down mgr daemons... 2023-12-10T23:38:07.368 DEBUG:tasks.ceph.cluster2.mgr.x:waiting for process to exit 2023-12-10T23:38:07.368 INFO:teuthology.orchestra.run:waiting for 300 2023-12-10T23:38:07.369 INFO:tasks.ceph.cluster2.mgr.x:Stopped 2023-12-10T23:38:07.369 INFO:teuthology.misc:Shutting down mon daemons... 2023-12-10T23:38:07.369 DEBUG:tasks.ceph.cluster2.mon.a:waiting for process to exit 2023-12-10T23:38:07.369 INFO:teuthology.orchestra.run:waiting for 300 2023-12-10T23:38:07.369 INFO:tasks.ceph.cluster2.mon.a:Stopped 2023-12-10T23:38:07.369 INFO:tasks.ceph:Checking cluster log for badness... 2023-12-10T23:38:07.370 DEBUG:teuthology.orchestra.run.smithi076:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2023-12-10T23:38:07.391 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-0 on ubuntu@smithi076.front.sepia.ceph.com 2023-12-10T23:38:07.391 DEBUG:teuthology.orchestra.run.smithi076:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-0 2023-12-10T23:38:07.723 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-1 on ubuntu@smithi076.front.sepia.ceph.com 2023-12-10T23:38:07.723 DEBUG:teuthology.orchestra.run.smithi076:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-1 2023-12-10T23:38:08.115 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-2 on ubuntu@smithi076.front.sepia.ceph.com 2023-12-10T23:38:08.116 DEBUG:teuthology.orchestra.run.smithi076:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-2 2023-12-10T23:38:08.389 INFO:tasks.ceph:Archiving mon data... 2023-12-10T23:38:08.391 DEBUG:teuthology.misc:Transferring archived files from smithi076:/var/lib/ceph/mon/cluster2-a to /home/teuthworker/archive/dis-2023-12-10_16:10:42-rbd-wip-dis-testing-1-distro-default-smithi/7486362/data/cluster2.mon.a.tgz 2023-12-10T23:38:08.391 DEBUG:teuthology.orchestra.run.smithi076:> mktemp 2023-12-10T23:38:08.396 INFO:teuthology.orchestra.run.smithi076.stdout:/tmp/tmp.GqJ5LPuGC9 2023-12-10T23:38:08.396 DEBUG:teuthology.orchestra.run.smithi076:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster2-a -- . > /tmp/tmp.GqJ5LPuGC9 2023-12-10T23:38:08.535 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod 0666 /tmp/tmp.GqJ5LPuGC9 2023-12-10T23:38:08.598 DEBUG:teuthology.orchestra.remote:smithi076:/tmp/tmp.GqJ5LPuGC9 is 412KB 2023-12-10T23:38:08.659 DEBUG:teuthology.orchestra.run.smithi076:> rm -fr /tmp/tmp.GqJ5LPuGC9 2023-12-10T23:38:08.665 INFO:tasks.ceph:Cleaning ceph cluster... 2023-12-10T23:38:08.665 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2023-12-10T23:38:08.668 DEBUG:teuthology.orchestra.run.smithi076:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2023-12-10T23:38:08.722 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi076 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' 2023-12-10T23:38:08.735 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-12-10T23:38:08.747 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2023-12-10T23:38:08.747 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2023-12-10T23:40:08.771 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-10T23:40:08.773 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi033 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json' 2023-12-10T23:40:08.788 INFO:teuthology.misc:Shutting down mds daemons... 2023-12-10T23:40:08.788 INFO:teuthology.misc:Shutting down osd daemons... 2023-12-10T23:40:08.789 DEBUG:tasks.ceph.cluster1.osd.0:waiting for process to exit 2023-12-10T23:40:08.789 INFO:teuthology.orchestra.run:waiting for 300 2023-12-10T23:40:08.793 INFO:tasks.ceph.cluster1.osd.0:Stopped 2023-12-10T23:40:08.793 DEBUG:tasks.ceph.cluster1.osd.1:waiting for process to exit 2023-12-10T23:40:08.793 INFO:teuthology.orchestra.run:waiting for 300 2023-12-10T23:40:08.793 INFO:tasks.ceph.cluster1.osd.1:Stopped 2023-12-10T23:40:08.794 DEBUG:tasks.ceph.cluster1.osd.2:waiting for process to exit 2023-12-10T23:40:08.794 INFO:teuthology.orchestra.run:waiting for 300 2023-12-10T23:40:08.794 INFO:tasks.ceph.cluster1.osd.2:Stopped 2023-12-10T23:40:08.794 INFO:teuthology.misc:Shutting down mgr daemons... 2023-12-10T23:40:08.794 DEBUG:tasks.ceph.cluster1.mgr.x:waiting for process to exit 2023-12-10T23:40:08.794 INFO:teuthology.orchestra.run:waiting for 300 2023-12-10T23:40:08.795 INFO:tasks.ceph.cluster1.mgr.x:Stopped 2023-12-10T23:40:08.795 INFO:teuthology.misc:Shutting down mon daemons... 2023-12-10T23:40:08.795 DEBUG:tasks.ceph.cluster1.mon.a:waiting for process to exit 2023-12-10T23:40:08.795 INFO:teuthology.orchestra.run:waiting for 300 2023-12-10T23:40:08.795 INFO:tasks.ceph.cluster1.mon.a:Stopped 2023-12-10T23:40:08.796 INFO:tasks.ceph:Checking cluster log for badness... 2023-12-10T23:40:08.796 DEBUG:teuthology.orchestra.run.smithi033:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2023-12-10T23:40:08.814 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-0 on ubuntu@smithi033.front.sepia.ceph.com 2023-12-10T23:40:08.814 DEBUG:teuthology.orchestra.run.smithi033:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-0 2023-12-10T23:40:08.979 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-1 on ubuntu@smithi033.front.sepia.ceph.com 2023-12-10T23:40:08.980 DEBUG:teuthology.orchestra.run.smithi033:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-1 2023-12-10T23:40:09.221 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-2 on ubuntu@smithi033.front.sepia.ceph.com 2023-12-10T23:40:09.222 DEBUG:teuthology.orchestra.run.smithi033:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-2 2023-12-10T23:40:09.349 INFO:tasks.ceph:Archiving mon data... 2023-12-10T23:40:09.350 DEBUG:teuthology.misc:Transferring archived files from smithi033:/var/lib/ceph/mon/cluster1-a to /home/teuthworker/archive/dis-2023-12-10_16:10:42-rbd-wip-dis-testing-1-distro-default-smithi/7486362/data/cluster1.mon.a.tgz 2023-12-10T23:40:09.350 DEBUG:teuthology.orchestra.run.smithi033:> mktemp 2023-12-10T23:40:09.356 INFO:teuthology.orchestra.run.smithi033.stdout:/tmp/tmp.zrVecsR9BM 2023-12-10T23:40:09.356 DEBUG:teuthology.orchestra.run.smithi033:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster1-a -- . > /tmp/tmp.zrVecsR9BM 2023-12-10T23:40:09.483 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 0666 /tmp/tmp.zrVecsR9BM 2023-12-10T23:40:09.548 DEBUG:teuthology.orchestra.remote:smithi033:/tmp/tmp.zrVecsR9BM is 328KB 2023-12-10T23:40:09.608 DEBUG:teuthology.orchestra.run.smithi033:> rm -fr /tmp/tmp.zrVecsR9BM 2023-12-10T23:40:09.613 INFO:tasks.ceph:Cleaning ceph cluster... 2023-12-10T23:40:09.613 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2023-12-10T23:40:09.659 DEBUG:teuthology.orchestra.run.smithi076:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2023-12-10T23:40:09.720 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2023-12-10T23:40:09.726 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2023-12-10T23:40:09.726 INFO:tasks.ceph:Archiving crash dumps... 2023-12-10T23:40:09.729 DEBUG:teuthology.misc:Transferring archived files from smithi033:/var/lib/ceph/crash to /home/teuthworker/archive/dis-2023-12-10_16:10:42-rbd-wip-dis-testing-1-distro-default-smithi/7486362/remote/smithi033/crash 2023-12-10T23:40:09.730 DEBUG:teuthology.orchestra.run.smithi033:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-12-10T23:40:09.775 DEBUG:teuthology.misc:Transferring archived files from smithi076:/var/lib/ceph/crash to /home/teuthworker/archive/dis-2023-12-10_16:10:42-rbd-wip-dis-testing-1-distro-default-smithi/7486362/remote/smithi076/crash 2023-12-10T23:40:09.776 DEBUG:teuthology.orchestra.run.smithi076:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-12-10T23:40:09.879 INFO:tasks.ceph:Compressing logs... 2023-12-10T23:40:09.880 DEBUG:teuthology.orchestra.run.smithi033:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-10T23:40:09.882 DEBUG:teuthology.orchestra.run.smithi076:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-10T23:40:09.900 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38953.log 2023-12-10T23:40:09.900 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1.audit.log 2023-12-10T23:40:09.901 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/cluster1-client.admin.38953.log: 0.0%gzip -- replaced with /var/log/ceph/cluster1-client.admin.38953.log.gz -5 2023-12-10T23:40:09.901 INFO:teuthology.orchestra.run.smithi033.stderr: --verbose -- /var/log/ceph/cluster1-client.admin.39194.log 2023-12-10T23:40:09.901 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/cluster1.audit.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39272.log 2023-12-10T23:40:09.903 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39056.log 2023-12-10T23:40:09.903 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/cluster1-client.admin.39194.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39194.log.gz 2023-12-10T23:40:09.903 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/cluster1-client.admin.39272.log: gzip -5 --verbose -- /var/log/ceph/cluster1-mon.a.log 2023-12-10T23:40:09.903 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39272.log.gz 2023-12-10T23:40:09.903 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/cluster1-client.admin.39056.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34592.log 2023-12-10T23:40:09.903 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39056.log.gz 2023-12-10T23:40:09.904 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/cluster1-mon.a.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39114.log 2023-12-10T23:40:09.905 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/cluster1-client.admin.34592.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39394.log 2023-12-10T23:40:09.906 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.34592.log.gz 2023-12-10T23:40:09.906 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/cluster1-client.admin.39114.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39114.log.gz 2023-12-10T23:40:09.906 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39004.log 2023-12-10T23:40:09.906 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39268.log 2023-12-10T23:40:09.906 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/cluster1-client.admin.39394.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39394.log.gz 2023-12-10T23:40:09.906 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/cluster1-client.admin.39004.log: 93.1% -- replaced with /var/log/ceph/cluster1.audit.log.gz 0.0% 2023-12-10T23:40:09.907 INFO:teuthology.orchestra.run.smithi033.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.39004.log.gz 2023-12-10T23:40:09.907 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39264.log 2023-12-10T23:40:09.907 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/cluster1-client.admin.39268.log: gzip -5 --verbose -- /var/log/ceph/cluster2-mgr.x.log 2023-12-10T23:40:09.907 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39268.log.gz 2023-12-10T23:40:09.907 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/cluster1-client.admin.39264.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39193.log 2023-12-10T23:40:09.907 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39264.log.gz 2023-12-10T23:40:09.910 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/cluster2-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34856.log 2023-12-10T23:40:09.910 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/cluster1-client.admin.39193.log: gzip -5 --verbose -- /var/log/ceph/cluster1-mgr.x.log 2023-12-10T23:40:09.910 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39193.log.gz 2023-12-10T23:40:09.913 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/cluster1-client.admin.34856.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.34856.log.gz 2023-12-10T23:40:09.913 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-osd.2.log 2023-12-10T23:40:09.914 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/cluster1-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34734.log 2023-12-10T23:40:09.914 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/cluster1-osd.2.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39368.log 2023-12-10T23:40:09.914 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-osd.0.log 2023-12-10T23:40:09.914 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/cluster1-client.admin.34734.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.34734.log.gz 2023-12-10T23:40:09.914 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/cluster1-client.admin.39368.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38978.log 2023-12-10T23:40:09.914 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39368.log.gz 2023-12-10T23:40:09.914 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/cluster1-osd.0.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39116.log 2023-12-10T23:40:09.915 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/cluster1-client.admin.38978.log: gzip -5 --verbose -- 0.0% /var/log/ceph/cluster1-client.admin.39030.log -- replaced with /var/log/ceph/cluster1-client.admin.38978.log.gz 2023-12-10T23:40:09.915 INFO:teuthology.orchestra.run.smithi033.stderr: 2023-12-10T23:40:09.918 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/cluster1-client.admin.39116.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39342.log 2023-12-10T23:40:09.919 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39116.log.gz 2023-12-10T23:40:09.919 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/cluster1-client.admin.39030.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39030.log.gz 2023-12-10T23:40:09.919 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34769.log 2023-12-10T23:40:09.919 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/cluster1-client.admin.39342.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40988.log 2023-12-10T23:40:09.919 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39342.log.gz 2023-12-10T23:40:09.919 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/cluster1-client.admin.34769.log: 0.0%gzip -- replaced with /var/log/ceph/cluster1-client.admin.34769.log.gz 2023-12-10T23:40:09.919 INFO:teuthology.orchestra.run.smithi033.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.admin.34686.log 2023-12-10T23:40:09.919 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/cluster1-client.admin.40988.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38892.log 2023-12-10T23:40:09.919 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.40988.log.gz 2023-12-10T23:40:09.920 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/cluster1-client.admin.34686.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39186.log 2023-12-10T23:40:09.920 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.34686.log.gz 2023-12-10T23:40:09.920 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/cluster1-client.admin.38892.log: gzip -5 --verbose -- /var/log/ceph/cluster1.log 2023-12-10T23:40:09.920 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.38892.log.gz 2023-12-10T23:40:09.920 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/cluster1-client.admin.39186.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39186.log.gz 2023-12-10T23:40:09.920 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39115.log 2023-12-10T23:40:09.920 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34710.log 2023-12-10T23:40:09.920 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/cluster1.log: /var/log/ceph/cluster1-client.admin.39115.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39082.log 2023-12-10T23:40:09.920 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39115.log.gz 2023-12-10T23:40:09.921 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/cluster1-client.admin.34710.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.34710.log.gz 2023-12-10T23:40:09.921 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.30961.log 2023-12-10T23:40:09.921 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/cluster1-client.admin.39082.log: gzip -5 --verbose -- /var/log/ceph/cluster1-osd.1.log 2023-12-10T23:40:09.921 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39082.log.gz 2023-12-10T23:40:09.921 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph.tmp-client.admin.30961.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.30961.log.gz 2023-12-10T23:40:09.922 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/cluster1-osd.1.log: 88.7% -- replaced with /var/log/ceph/cluster1.log.gz 2023-12-10T23:40:09.939 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65882.log 2023-12-10T23:40:09.939 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60368.log 2023-12-10T23:40:09.940 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.65882.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42764.log 2023-12-10T23:40:09.940 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.60368.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.65882.log.gz 2023-12-10T23:40:09.940 INFO:teuthology.orchestra.run.smithi076.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60368.log.gz 2023-12-10T23:40:09.940 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56871.log 2023-12-10T23:40:09.941 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.39969.log 2023-12-10T23:40:09.941 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.56871.log: /var/log/ceph/cluster2-client.mirror.42764.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65836.log 2023-12-10T23:40:09.942 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.0.39969.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67009.log 2023-12-10T23:40:09.942 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.65836.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76406.log 2023-12-10T23:40:09.942 INFO:teuthology.orchestra.run.smithi076.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.65836.log.gz 2023-12-10T23:40:09.943 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.67009.log: 92.4%gzip -- replaced with /var/log/ceph/cluster1-client.mirror.56871.log.gz 2023-12-10T23:40:09.943 INFO:teuthology.orchestra.run.smithi076.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42608.log 2023-12-10T23:40:09.944 INFO:teuthology.orchestra.run.smithi076.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.67009.log.gz 2023-12-10T23:40:09.944 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.76406.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75284.log 2023-12-10T23:40:09.944 INFO:teuthology.orchestra.run.smithi076.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.76406.log.gz 2023-12-10T23:40:09.945 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.78107.log 2023-12-10T23:40:09.945 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.42608.log: /var/log/ceph/cluster2-client.mirror.75284.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40434.log 2023-12-10T23:40:09.945 INFO:teuthology.orchestra.run.smithi076.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.75284.log.gz 2023-12-10T23:40:09.945 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.admin.78107.log: 50.8%gzip -- replaced with /var/log/ceph/cluster1-client.admin.78107.log.gz 2023-12-10T23:40:09.945 INFO:teuthology.orchestra.run.smithi076.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64379.log 2023-12-10T23:40:09.945 INFO:teuthology.orchestra.run.smithi076.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.42608.log.gz 2023-12-10T23:40:09.949 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.admin.40434.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74160.log 2023-12-10T23:40:09.949 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.40434.log.gz 2023-12-10T23:40:09.949 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.64379.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.0.39969.log.gz 2023-12-10T23:40:09.949 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59008.log 2023-12-10T23:40:09.950 INFO:teuthology.orchestra.run.smithi076.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.64379.log.gz 2023-12-10T23:40:09.950 INFO:teuthology.orchestra.run.smithi076.stderr:gzip/var/log/ceph/cluster2-client.mirror.74160.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42169.log 2023-12-10T23:40:09.950 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.59008.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71065.log 2023-12-10T23:40:09.950 INFO:teuthology.orchestra.run.smithi076.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.74160.log.gz 2023-12-10T23:40:09.950 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.42169.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70277.log 2023-12-10T23:40:09.950 INFO:teuthology.orchestra.run.smithi076.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.42764.log.gz 2023-12-10T23:40:09.950 INFO:teuthology.orchestra.run.smithi076.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.42169.log.gz 2023-12-10T23:40:09.950 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.71065.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61090.log 2023-12-10T23:40:09.950 INFO:teuthology.orchestra.run.smithi076.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59008.log.gz 2023-12-10T23:40:09.951 INFO:teuthology.orchestra.run.smithi076.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.71065.log.gz 2023-12-10T23:40:09.951 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.70277.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49204.log 2023-12-10T23:40:09.951 INFO:teuthology.orchestra.run.smithi076.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.70277.log.gz 2023-12-10T23:40:09.951 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.61090.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59422.log 2023-12-10T23:40:09.951 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.49204.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61090.log.gz 2023-12-10T23:40:09.951 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65020.log 2023-12-10T23:40:09.951 INFO:teuthology.orchestra.run.smithi076.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.49204.log.gz 2023-12-10T23:40:09.951 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.59422.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57095.log 2023-12-10T23:40:09.951 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.65020.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54164.log 2023-12-10T23:40:09.952 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.57095.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.53058.log 2023-12-10T23:40:09.956 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.54164.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.59422.log.gz 2023-12-10T23:40:09.956 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67236.log 2023-12-10T23:40:09.956 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.2.53058.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76364.log 2023-12-10T23:40:09.956 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.67236.log: gzip -5 --verbose 92.0% -- -- replaced with /var/log/ceph/cluster2-client.mirror.57095.log.gz /var/log/ceph/cluster2-client.mirror.67878.log 2023-12-10T23:40:09.957 INFO:teuthology.orchestra.run.smithi076.stderr: 2023-12-10T23:40:09.957 INFO:teuthology.orchestra.run.smithi076.stderr: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.65020.log.gz 2023-12-10T23:40:09.957 INFO:teuthology.orchestra.run.smithi076.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.67236.log.gz 2023-12-10T23:40:09.957 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.76364.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54164.log.gz 2023-12-10T23:40:09.957 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60598.log 2023-12-10T23:40:09.957 INFO:teuthology.orchestra.run.smithi076.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.76364.log.gz 2023-12-10T23:40:09.957 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48590.log 2023-12-10T23:40:09.957 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.67878.log: /var/log/ceph/cluster1-client.mirror.60598.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73787.log 2023-12-10T23:40:09.957 INFO:teuthology.orchestra.run.smithi076.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.67878.log.gz 2023-12-10T23:40:09.958 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.48590.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.60598.log.gz 2023-12-10T23:40:09.958 INFO:teuthology.orchestra.run.smithi076.stderr: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48590.log.gz 2023-12-10T23:40:09.958 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46034.log 2023-12-10T23:40:09.958 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.77777.log 2023-12-10T23:40:09.958 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.73787.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48497.log 2023-12-10T23:40:09.958 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.46034.log: /var/log/ceph/cluster1-client.admin.77777.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53617.log 2023-12-10T23:40:09.958 INFO:teuthology.orchestra.run.smithi076.stderr: 70.6% -- replaced with /var/log/ceph/cluster1-client.admin.77777.log.gz 2023-12-10T23:40:09.958 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.48497.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59522.log 2023-12-10T23:40:09.958 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.53617.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.73787.log.gz 2023-12-10T23:40:09.958 INFO:teuthology.orchestra.run.smithi076.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48497.log.gz 2023-12-10T23:40:09.959 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.52067.log 2023-12-10T23:40:09.959 INFO:teuthology.orchestra.run.smithi076.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53617.log.gz 2023-12-10T23:40:09.959 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76529.log 2023-12-10T23:40:09.959 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.59522.log: /var/log/ceph/cluster2-client.mirror.2.52067.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45370.log 2023-12-10T23:40:09.959 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.76529.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.59522.log.gz 2023-12-10T23:40:09.959 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77067.log 2023-12-10T23:40:09.959 INFO:teuthology.orchestra.run.smithi076.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.76529.log.gz 2023-12-10T23:40:09.959 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.45370.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47275.log 2023-12-10T23:40:09.959 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78694.log 2023-12-10T23:40:09.959 INFO:teuthology.orchestra.run.smithi076.stderr: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.45370.log.gz 2023-12-10T23:40:09.960 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.77067.log: /var/log/ceph/cluster2-client.mirror.47275.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61113.log 2023-12-10T23:40:09.965 INFO:teuthology.orchestra.run.smithi076.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.77067.log.gz 2023-12-10T23:40:09.965 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.78694.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63256.log 2023-12-10T23:40:09.965 INFO:teuthology.orchestra.run.smithi076.stderr: 9.2% -- replaced with /var/log/ceph/cluster2-client.mirror.78694.log.gz 2023-12-10T23:40:09.965 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.61113.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64197.log 2023-12-10T23:40:09.965 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.63256.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.61113.log.gz 2023-12-10T23:40:09.965 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76820.log 2023-12-10T23:40:09.965 INFO:teuthology.orchestra.run.smithi076.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.63256.log.gz 2023-12-10T23:40:09.965 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58650.log 2023-12-10T23:40:09.966 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.64197.log: /var/log/ceph/cluster1-client.mirror.76820.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52156.log 2023-12-10T23:40:09.966 INFO:teuthology.orchestra.run.smithi076.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.64197.log.gz 2023-12-10T23:40:09.966 INFO:teuthology.orchestra.run.smithi076.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.76820.log.gz 2023-12-10T23:40:09.966 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.58650.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55499.log 2023-12-10T23:40:09.966 INFO:teuthology.orchestra.run.smithi076.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58650.log.gz 2023-12-10T23:40:09.966 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.52156.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69557.log 2023-12-10T23:40:09.966 INFO:teuthology.orchestra.run.smithi076.stderr: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.52156.log.gz 2023-12-10T23:40:09.966 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.55499.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49921.log 2023-12-10T23:40:09.966 INFO:teuthology.orchestra.run.smithi076.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.46034.log.gz 2023-12-10T23:40:09.967 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.69557.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72989.log 2023-12-10T23:40:09.967 INFO:teuthology.orchestra.run.smithi076.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.69557.log.gz 2023-12-10T23:40:09.967 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.49921.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76320.log 2023-12-10T23:40:09.967 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.72989.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65080.log 2023-12-10T23:40:09.967 INFO:teuthology.orchestra.run.smithi076.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49921.log.gz 2023-12-10T23:40:09.967 INFO:teuthology.orchestra.run.smithi076.stderr: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.72989.log.gz 2023-12-10T23:40:09.967 INFO:teuthology.orchestra.run.smithi076.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47275.log.gz 2023-12-10T23:40:09.967 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.76320.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78141.log 2023-12-10T23:40:09.967 INFO:teuthology.orchestra.run.smithi076.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55499.log.gz 2023-12-10T23:40:09.967 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.65080.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50275.log 2023-12-10T23:40:09.968 INFO:teuthology.orchestra.run.smithi076.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.76320.log.gz 2023-12-10T23:40:09.968 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.78141.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61509.log 2023-12-10T23:40:09.968 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78141.log.gz 2023-12-10T23:40:09.968 INFO:teuthology.orchestra.run.smithi076.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.65080.log.gz 2023-12-10T23:40:09.968 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.50275.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64022.log 2023-12-10T23:40:09.968 INFO:teuthology.orchestra.run.smithi076.stderr: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.50275.log.gz 2023-12-10T23:40:09.968 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.61509.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67050.log 2023-12-10T23:40:09.968 INFO:teuthology.orchestra.run.smithi076.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.2.52067.log.gz 2023-12-10T23:40:09.968 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.64022.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50042.log 2023-12-10T23:40:09.968 INFO:teuthology.orchestra.run.smithi076.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.61509.log.gz 2023-12-10T23:40:09.969 INFO:teuthology.orchestra.run.smithi076.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.64022.log.gz 2023-12-10T23:40:09.969 INFO:teuthology.orchestra.run.smithi076.stderr:gzip/var/log/ceph/cluster2-client.mirror.67050.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74849.log 2023-12-10T23:40:09.969 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.50042.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72781.log 2023-12-10T23:40:09.969 INFO:teuthology.orchestra.run.smithi076.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.67050.log.gz 2023-12-10T23:40:09.969 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.74849.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67174.log 2023-12-10T23:40:09.969 INFO:teuthology.orchestra.run.smithi076.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.74849.log.gz 2023-12-10T23:40:09.978 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.72781.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65123.log 2023-12-10T23:40:09.978 INFO:teuthology.orchestra.run.smithi076.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.72781.log.gz 2023-12-10T23:40:09.978 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.67174.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74456.log 2023-12-10T23:40:09.978 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.65123.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.67174.log.gz 2023-12-10T23:40:09.978 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56025.log 2023-12-10T23:40:09.978 INFO:teuthology.orchestra.run.smithi076.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.65123.log.gz 2023-12-10T23:40:09.978 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.74456.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55900.log 2023-12-10T23:40:09.978 INFO:teuthology.orchestra.run.smithi076.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.74456.log.gz 2023-12-10T23:40:09.979 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45666.log 2023-12-10T23:40:09.979 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.56025.log: /var/log/ceph/cluster1-client.mirror.55900.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43797.log 2023-12-10T23:40:09.979 INFO:teuthology.orchestra.run.smithi076.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.56025.log.gz 2023-12-10T23:40:09.979 INFO:teuthology.orchestra.run.smithi076.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55900.log.gz 2023-12-10T23:40:09.979 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.admin.45666.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48867.log 2023-12-10T23:40:09.979 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69962.log 2023-12-10T23:40:09.979 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.43797.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.45666.log.gz 2023-12-10T23:40:09.979 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.48867.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55563.log 2023-12-10T23:40:09.979 INFO:teuthology.orchestra.run.smithi076.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.43797.log.gz 2023-12-10T23:40:09.979 INFO:teuthology.orchestra.run.smithi076.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48867.log.gz 2023-12-10T23:40:09.980 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.69962.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68689.log 2023-12-10T23:40:09.980 INFO:teuthology.orchestra.run.smithi076.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.69962.log.gz 2023-12-10T23:40:09.980 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.55563.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78820.log 2023-12-10T23:40:09.980 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.68689.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56429.log 2023-12-10T23:40:09.980 INFO:teuthology.orchestra.run.smithi076.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.68689.log.gz 2023-12-10T23:40:09.980 INFO:teuthology.orchestra.run.smithi076.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.55563.log.gz 2023-12-10T23:40:09.980 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.admin.78820.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60477.log 2023-12-10T23:40:09.980 INFO:teuthology.orchestra.run.smithi076.stderr: 9.2% -- replaced with /var/log/ceph/cluster2-client.admin.78820.log.gz 2023-12-10T23:40:09.980 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63563.log 2023-12-10T23:40:09.980 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.56429.log: /var/log/ceph/cluster2-client.mirror.60477.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61665.log 2023-12-10T23:40:09.981 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.63563.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.60477.log.gz 2023-12-10T23:40:09.981 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78645.log 2023-12-10T23:40:09.981 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40852.log 2023-12-10T23:40:09.981 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.61665.log: /var/log/ceph/cluster2-client.mirror.78645.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40238.log 2023-12-10T23:40:09.981 INFO:teuthology.orchestra.run.smithi076.stderr: 9.2% -- replaced with /var/log/ceph/cluster2-client.mirror.78645.log.gz 2023-12-10T23:40:09.981 INFO:teuthology.orchestra.run.smithi076.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56429.log.gz 2023-12-10T23:40:09.981 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.40852.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51051.log 2023-12-10T23:40:09.981 INFO:teuthology.orchestra.run.smithi076.stderr: 53.8% -- replaced with /var/log/ceph/cluster2-client.mirror.40852.log.gz 2023-12-10T23:40:09.981 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52421.log 2023-12-10T23:40:09.981 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.admin.40238.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.63563.log.gz 2023-12-10T23:40:09.982 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.51051.log: 80.1% -- replaced with /var/log/ceph/cluster1-client.admin.40238.log.gz 2023-12-10T23:40:09.982 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47295.log 2023-12-10T23:40:09.982 INFO:teuthology.orchestra.run.smithi076.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.61665.log.gz 2023-12-10T23:40:09.982 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.52421.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64399.log 2023-12-10T23:40:09.982 INFO:teuthology.orchestra.run.smithi076.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51051.log.gz 2023-12-10T23:40:09.982 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.47295.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69985.log 2023-12-10T23:40:09.982 INFO:teuthology.orchestra.run.smithi076.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52421.log.gz 2023-12-10T23:40:09.982 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.64399.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55095.log 2023-12-10T23:40:09.982 INFO:teuthology.orchestra.run.smithi076.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.64399.log.gz 2023-12-10T23:40:09.982 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.69985.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44411.log 2023-12-10T23:40:09.983 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.55095.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.69985.log.gz 2023-12-10T23:40:09.983 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75995.log 2023-12-10T23:40:09.983 INFO:teuthology.orchestra.run.smithi076.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55095.log.gz 2023-12-10T23:40:09.983 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.44411.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48830.log 2023-12-10T23:40:09.983 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.75995.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.44411.log.gz 2023-12-10T23:40:09.983 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77487.log 2023-12-10T23:40:09.983 INFO:teuthology.orchestra.run.smithi076.stderr: 87.1% -- replaced with /var/log/ceph/cluster2-client.mirror.75995.log.gz 2023-12-10T23:40:09.983 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.48830.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52994.log 2023-12-10T23:40:09.983 INFO:teuthology.orchestra.run.smithi076.stderr: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48830.log.gz 2023-12-10T23:40:09.983 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.77487.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63302.log 2023-12-10T23:40:09.983 INFO:teuthology.orchestra.run.smithi076.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.77487.log.gz 2023-12-10T23:40:09.984 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.52994.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53362.log 2023-12-10T23:40:09.984 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.63302.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74722.log 2023-12-10T23:40:09.984 INFO:teuthology.orchestra.run.smithi076.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.63302.log.gz 2023-12-10T23:40:09.984 INFO:teuthology.orchestra.run.smithi076.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52994.log.gz 2023-12-10T23:40:09.984 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46077.log 2023-12-10T23:40:09.984 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.53362.log: /var/log/ceph/cluster2-client.mirror.74722.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60869.log 2023-12-10T23:40:09.984 INFO:teuthology.orchestra.run.smithi076.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53362.log.gz 2023-12-10T23:40:09.992 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.46077.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55007.log 2023-12-10T23:40:09.992 INFO:teuthology.orchestra.run.smithi076.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.74722.log.gz 2023-12-10T23:40:09.992 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.60869.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45858.log 2023-12-10T23:40:09.992 INFO:teuthology.orchestra.run.smithi076.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.60869.log.gz 2023-12-10T23:40:09.992 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.55007.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66389.log 2023-12-10T23:40:09.992 INFO:teuthology.orchestra.run.smithi076.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.47295.log.gz 2023-12-10T23:40:09.992 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.45858.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46385.log 2023-12-10T23:40:09.992 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.66389.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55652.log 2023-12-10T23:40:09.993 INFO:teuthology.orchestra.run.smithi076.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45858.log.gz/var/log/ceph/cluster1-client.mirror.46385.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.66389.log.gz 2023-12-10T23:40:09.993 INFO:teuthology.orchestra.run.smithi076.stderr: 2023-12-10T23:40:09.993 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76098.log 2023-12-10T23:40:09.993 INFO:teuthology.orchestra.run.smithi076.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.55007.log.gz 2023-12-10T23:40:09.993 INFO:teuthology.orchestra.run.smithi076.stderr: 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.46385.log.gz 2023-12-10T23:40:09.993 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75580.log 2023-12-10T23:40:09.993 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.55652.log: /var/log/ceph/cluster2-client.mirror.76098.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51893.log 2023-12-10T23:40:09.993 INFO:teuthology.orchestra.run.smithi076.stderr: 80.0% -- replaced with /var/log/ceph/cluster2-client.mirror.76098.log.gz 2023-12-10T23:40:09.993 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.75580.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55652.log.gz 2023-12-10T23:40:09.993 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74139.log 2023-12-10T23:40:09.994 INFO:teuthology.orchestra.run.smithi076.stderr: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.75580.log.gz 2023-12-10T23:40:09.994 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.51893.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43199.log 2023-12-10T23:40:09.994 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51071.log 2023-12-10T23:40:09.994 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.74139.log: /var/log/ceph/cluster2-client.mirror.43199.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78162.log 2023-12-10T23:40:09.994 INFO:teuthology.orchestra.run.smithi076.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.74139.log.gz 2023-12-10T23:40:09.994 INFO:teuthology.orchestra.run.smithi076.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.46077.log.gz 2023-12-10T23:40:09.994 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.51071.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.51071.log.gz 2023-12-10T23:40:09.994 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68127.log 2023-12-10T23:40:09.994 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.78162.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66182.log 2023-12-10T23:40:09.994 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78162.log.gz 2023-12-10T23:40:09.995 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.68127.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60575.log 2023-12-10T23:40:09.995 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.66182.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.68127.log.gz 2023-12-10T23:40:09.995 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67029.log 2023-12-10T23:40:09.995 INFO:teuthology.orchestra.run.smithi076.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.66182.log.gz 2023-12-10T23:40:09.995 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.60575.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.75471.log 2023-12-10T23:40:09.995 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.67029.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60575.log.gz 2023-12-10T23:40:09.995 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78459.log 2023-12-10T23:40:09.995 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.0.75471.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65192.log 2023-12-10T23:40:09.995 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.78459.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63450.log 2023-12-10T23:40:09.996 INFO:teuthology.orchestra.run.smithi076.stderr: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.78459.log.gz 2023-12-10T23:40:09.996 INFO:teuthology.orchestra.run.smithi076.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.67029.log.gz 2023-12-10T23:40:09.996 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.65192.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.65192.log.gz 2023-12-10T23:40:09.996 INFO:teuthology.orchestra.run.smithi076.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43199.log.gz 2023-12-10T23:40:09.996 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66843.log 2023-12-10T23:40:09.996 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.63450.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44723.log 2023-12-10T23:40:09.996 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.66843.log: gzip -5 --verbose 87.4% -- -- replaced with /var/log/ceph/cluster1-client.mirror.63450.log.gz /var/log/ceph/cluster2-client.mirror.68619.log 2023-12-10T23:40:09.996 INFO:teuthology.orchestra.run.smithi076.stderr: 2023-12-10T23:40:09.996 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.admin.44723.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58008.log 2023-12-10T23:40:09.996 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.68619.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.44723.log.gz 2023-12-10T23:40:09.997 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.34914.log 2023-12-10T23:40:09.997 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49419.log 2023-12-10T23:40:09.997 INFO:teuthology.orchestra.run.smithi076.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.68619.log.gz 2023-12-10T23:40:09.997 INFO:teuthology.orchestra.run.smithi076.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.66843.log.gz 2023-12-10T23:40:09.997 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.58008.log: /var/log/ceph/cluster2-client.admin.34914.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.34914.log.gz 2023-12-10T23:40:09.997 INFO:teuthology.orchestra.run.smithi076.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58008.log.gz 2023-12-10T23:40:09.997 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59705.log 2023-12-10T23:40:09.997 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.49419.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.53149.log 2023-12-10T23:40:10.002 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.59705.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67112.log 2023-12-10T23:40:10.002 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.3.53149.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.59705.log.gz 2023-12-10T23:40:10.002 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58345.log 2023-12-10T23:40:10.002 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61285.log 2023-12-10T23:40:10.002 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.58345.log: /var/log/ceph/cluster2-client.mirror.67112.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.67112.log.gz 2023-12-10T23:40:10.002 INFO:teuthology.orchestra.run.smithi076.stderr: 88.9%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76755.log 2023-12-10T23:40:10.002 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.58345.log.gz 2023-12-10T23:40:10.003 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.61285.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61285.log.gz 2023-12-10T23:40:10.003 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59142.log 2023-12-10T23:40:10.003 INFO:teuthology.orchestra.run.smithi076.stderr: 94.3%/var/log/ceph/cluster1-client.mirror.76755.log: -- replaced with /var/log/ceph/cluster2-client.mirror.2.53058.log.gzgzip 2023-12-10T23:40:10.003 INFO:teuthology.orchestra.run.smithi076.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50834.log 2023-12-10T23:40:10.003 INFO:teuthology.orchestra.run.smithi076.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.76755.log.gz 2023-12-10T23:40:10.003 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.59142.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48627.log 2023-12-10T23:40:10.003 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.50834.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47037.log 2023-12-10T23:40:10.003 INFO:teuthology.orchestra.run.smithi076.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.59142.log.gz 2023-12-10T23:40:10.003 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.48627.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50834.log.gz 2023-12-10T23:40:10.003 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40408.log 2023-12-10T23:40:10.004 INFO:teuthology.orchestra.run.smithi076.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48627.log.gz 2023-12-10T23:40:10.004 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.47037.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38996.log 2023-12-10T23:40:10.004 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.admin.40408.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50992.log 2023-12-10T23:40:10.004 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.40408.log.gz 2023-12-10T23:40:10.004 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.admin.38996.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.38996.log.gz 2023-12-10T23:40:10.004 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46931.log 2023-12-10T23:40:10.004 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.50992.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.74005.log 2023-12-10T23:40:10.004 INFO:teuthology.orchestra.run.smithi076.stderr: 87.7%/var/log/ceph/cluster2-client.mirror.46931.log: -- replaced with /var/log/ceph/cluster2-client.mirror.50992.log.gz 2023-12-10T23:40:10.004 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47510.log 2023-12-10T23:40:10.005 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.3.74005.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.47037.log.gz 2023-12-10T23:40:10.005 INFO:teuthology.orchestra.run.smithi076.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46931.log.gz 2023-12-10T23:40:10.005 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42674.log 2023-12-10T23:40:10.006 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.47510.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60455.log 2023-12-10T23:40:10.006 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.42674.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.42674.log.gz 2023-12-10T23:40:10.006 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39726.log 2023-12-10T23:40:10.008 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.60455.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60455.log.gz 2023-12-10T23:40:10.009 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74869.log 2023-12-10T23:40:10.009 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.admin.39726.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39726.log.gz 2023-12-10T23:40:10.010 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59959.log 2023-12-10T23:40:10.011 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.74869.log: 87.1% -- replaced with /var/log/ceph/cluster2-client.mirror.74869.log.gz 2023-12-10T23:40:10.011 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58152.log 2023-12-10T23:40:10.012 INFO:teuthology.orchestra.run.smithi076.stderr: 93.5%/var/log/ceph/cluster2-client.mirror.59959.log: -- replaced with /var/log/ceph/cluster1-client.mirror.47510.log.gz 2023-12-10T23:40:10.012 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64512.log 2023-12-10T23:40:10.013 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.58152.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.58152.log.gz 2023-12-10T23:40:10.013 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76689.log 2023-12-10T23:40:10.014 INFO:teuthology.orchestra.run.smithi076.stderr: 92.3%/var/log/ceph/cluster2-client.mirror.64512.log: -- replaced with /var/log/ceph/cluster2-client.mirror.59959.log.gz 2023-12-10T23:40:10.014 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49686.log 2023-12-10T23:40:10.015 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.76689.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.76689.log.gz 2023-12-10T23:40:10.016 INFO:teuthology.orchestra.run.smithi076.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.64512.log.gz 2023-12-10T23:40:10.016 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64712.log 2023-12-10T23:40:10.016 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.49686.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66223.log 2023-12-10T23:40:10.017 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.64712.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.64712.log.gz 2023-12-10T23:40:10.017 INFO:teuthology.orchestra.run.smithi076.stderr: 94.6% -- replaced with /var/log/ceph/cluster1-client.mirror.49686.log.gz 2023-12-10T23:40:10.017 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.54854.log 2023-12-10T23:40:10.018 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.66223.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78531.log 2023-12-10T23:40:10.019 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.1.54854.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.66223.log.gz 2023-12-10T23:40:10.022 INFO:teuthology.orchestra.run.smithi076.stderr: 95.4% -- replaced with /var/log/ceph/cluster2-client.mirror.0.75471.log.gz 2023-12-10T23:40:10.022 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78520.log 2023-12-10T23:40:10.023 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76508.log 2023-12-10T23:40:10.023 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55323.log 2023-12-10T23:40:10.023 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57896.log 2023-12-10T23:40:10.024 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75113.log 2023-12-10T23:40:10.024 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.78531.log: /var/log/ceph/cluster2-client.mirror.78520.log: 10.2% -- replaced with /var/log/ceph/cluster2-client.mirror.78531.log.gz 2023-12-10T23:40:10.024 INFO:teuthology.orchestra.run.smithi076.stderr: 9.2% -- replaced with /var/log/ceph/cluster2-client.mirror.78520.log.gz 2023-12-10T23:40:10.025 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.76508.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57556.log 2023-12-10T23:40:10.025 INFO:teuthology.orchestra.run.smithi076.stderr: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.76508.log.gz 2023-12-10T23:40:10.026 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.75113.log: /var/log/ceph/cluster2-client.mirror.55323.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.55323.log.gz 2023-12-10T23:40:10.027 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.57896.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.57896.log.gz 2023-12-10T23:40:10.029 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43821.log 2023-12-10T23:40:10.032 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.57556.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57556.log.gz 2023-12-10T23:40:10.032 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53263.log 2023-12-10T23:40:10.033 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.43821.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43821.log.gz 2023-12-10T23:40:10.033 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46865.log 2023-12-10T23:40:10.035 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.53263.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.53263.log.gz 2023-12-10T23:40:10.035 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76157.log 2023-12-10T23:40:10.040 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.46865.log: 94.2% -- replaced with /var/log/ceph/cluster2-client.mirror.1.54854.log.gz 2023-12-10T23:40:10.041 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48409.log 2023-12-10T23:40:10.041 INFO:teuthology.orchestra.run.smithi076.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.46865.log.gz 2023-12-10T23:40:10.041 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.76157.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42849.log 2023-12-10T23:40:10.042 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.48409.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48409.log.gz 2023-12-10T23:40:10.043 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63427.log 2023-12-10T23:40:10.044 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.42849.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.42849.log.gz 2023-12-10T23:40:10.044 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71515.log 2023-12-10T23:40:10.045 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.63427.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.63427.log.gz 2023-12-10T23:40:10.045 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45523.log 2023-12-10T23:40:10.046 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.71515.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71515.log.gz 2023-12-10T23:40:10.046 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47596.log 2023-12-10T23:40:10.049 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.45523.log: 95.0% -- replaced with /var/log/ceph/cluster1-client.mirror.3.74005.log.gz 2023-12-10T23:40:10.052 INFO:teuthology.orchestra.run.smithi076.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.45523.log.gz 2023-12-10T23:40:10.052 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63365.log 2023-12-10T23:40:10.053 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77046.log 2023-12-10T23:40:10.054 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.47596.log: /var/log/ceph/cluster2-client.mirror.63365.log: 88.4% 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63365.log.gz 2023-12-10T23:40:10.054 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.47596.log.gz 2023-12-10T23:40:10.054 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45155.log 2023-12-10T23:40:10.055 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58696.log 2023-12-10T23:40:10.055 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.77046.log: /var/log/ceph/cluster2-client.mirror.45155.log: 87.1% -- replaced with /var/log/ceph/cluster2-client.mirror.77046.log.gz 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.45155.log.gz 2023-12-10T23:40:10.055 INFO:teuthology.orchestra.run.smithi076.stderr: 2023-12-10T23:40:10.056 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64243.log 2023-12-10T23:40:10.056 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70457.log 2023-12-10T23:40:10.057 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.58696.log: /var/log/ceph/cluster1-client.mirror.64243.log: 87.7% 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.58696.log.gz 2023-12-10T23:40:10.057 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.64243.log.gz 2023-12-10T23:40:10.057 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52563.log 2023-12-10T23:40:10.058 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.70457.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70457.log.gz 2023-12-10T23:40:10.058 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40460.log 2023-12-10T23:40:10.059 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63473.log 2023-12-10T23:40:10.059 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.52563.log: /var/log/ceph/cluster2-client.admin.40460.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52563.log.gz 2023-12-10T23:40:10.060 INFO:teuthology.orchestra.run.smithi076.stderr: 79.8% -- replaced with /var/log/ceph/cluster2-client.admin.40460.log.gz 2023-12-10T23:40:10.060 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60237.log 2023-12-10T23:40:10.060 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53246.log 2023-12-10T23:40:10.060 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.63473.log: /var/log/ceph/cluster2-client.mirror.60237.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.63473.log.gz 2023-12-10T23:40:10.061 INFO:teuthology.orchestra.run.smithi076.stderr: 88.5%gzip -- replaced with /var/log/ceph/cluster2-client.mirror.60237.log.gz 2023-12-10T23:40:10.061 INFO:teuthology.orchestra.run.smithi076.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56559.log 2023-12-10T23:40:10.061 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.53246.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.53246.log.gz 2023-12-10T23:40:10.062 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61999.log 2023-12-10T23:40:10.062 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42743.log 2023-12-10T23:40:10.063 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.56559.log: /var/log/ceph/cluster1-client.mirror.61999.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.56559.log.gz 2023-12-10T23:40:10.063 INFO:teuthology.orchestra.run.smithi076.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.61999.log.gz 2023-12-10T23:40:10.063 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77794.log 2023-12-10T23:40:10.064 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40018.log 2023-12-10T23:40:10.064 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.42743.log: /var/log/ceph/cluster1-client.mirror.77794.log: 68.0% -- replaced with /var/log/ceph/cluster1-client.mirror.77794.log.gz 2023-12-10T23:40:10.064 INFO:teuthology.orchestra.run.smithi076.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.42743.log.gz 2023-12-10T23:40:10.065 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43219.log 2023-12-10T23:40:10.065 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53876.log 2023-12-10T23:40:10.065 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.40018.log: /var/log/ceph/cluster1-client.mirror.43219.log: 81.6% -- replaced with /var/log/ceph/cluster1-client.mirror.40018.log.gz 2023-12-10T23:40:10.066 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54287.log 2023-12-10T23:40:10.067 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.53876.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53876.log.gz 2023-12-10T23:40:10.067 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78234.log 2023-12-10T23:40:10.068 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.54287.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54287.log.gz 2023-12-10T23:40:10.068 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51636.log 2023-12-10T23:40:10.069 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.78234.log: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.78234.log.gz 2023-12-10T23:40:10.069 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74808.log 2023-12-10T23:40:10.071 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.51636.log: 93.4% 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51636.log.gz 2023-12-10T23:40:10.071 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.43219.log.gz 2023-12-10T23:40:10.071 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47636.log 2023-12-10T23:40:10.072 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61752.log 2023-12-10T23:40:10.072 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.74808.log: /var/log/ceph/cluster2-client.mirror.47636.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.74808.log.gz 2023-12-10T23:40:10.073 INFO:teuthology.orchestra.run.smithi076.stderr: 88.4%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73603.log 2023-12-10T23:40:10.073 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.47636.log.gz 2023-12-10T23:40:10.086 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.61752.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64043.log 2023-12-10T23:40:10.086 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.73603.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.73603.log.gz 2023-12-10T23:40:10.087 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50876.log 2023-12-10T23:40:10.087 INFO:teuthology.orchestra.run.smithi076.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61752.log.gz 2023-12-10T23:40:10.089 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.64043.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.64043.log.gz 2023-12-10T23:40:10.089 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46491.log 2023-12-10T23:40:10.090 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58543.log 2023-12-10T23:40:10.090 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.50876.log: /var/log/ceph/cluster2-client.mirror.46491.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50876.log.gz 2023-12-10T23:40:10.091 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53594.log 2023-12-10T23:40:10.091 INFO:teuthology.orchestra.run.smithi076.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46491.log.gz 2023-12-10T23:40:10.091 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.58543.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58543.log.gz 2023-12-10T23:40:10.092 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76115.log 2023-12-10T23:40:10.092 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63323.log 2023-12-10T23:40:10.093 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.53594.log: /var/log/ceph/cluster2-client.mirror.76115.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.53594.log.gz 2023-12-10T23:40:10.093 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73767.log 2023-12-10T23:40:10.094 INFO:teuthology.orchestra.run.smithi076.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.76115.log.gz 2023-12-10T23:40:10.095 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.63323.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63323.log.gz 2023-12-10T23:40:10.096 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54966.log 2023-12-10T23:40:10.096 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64287.log 2023-12-10T23:40:10.097 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.73767.log: /var/log/ceph/cluster1-client.mirror.54966.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.73767.log.gz 2023-12-10T23:40:10.097 INFO:teuthology.orchestra.run.smithi076.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.54966.log.gz 2023-12-10T23:40:10.097 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59682.log 2023-12-10T23:40:10.099 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.64287.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64287.log.gz 2023-12-10T23:40:10.099 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58832.log 2023-12-10T23:40:10.100 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68771.log 2023-12-10T23:40:10.100 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.59682.log: /var/log/ceph/cluster2-client.mirror.58832.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59682.log.gz 2023-12-10T23:40:10.101 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44001.log 2023-12-10T23:40:10.101 INFO:teuthology.orchestra.run.smithi076.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.58832.log.gz 2023-12-10T23:40:10.103 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.68771.log: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.68771.log.gz 2023-12-10T23:40:10.103 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46426.log 2023-12-10T23:40:10.104 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57714.log 2023-12-10T23:40:10.104 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.44001.log: /var/log/ceph/cluster1-client.mirror.46426.log: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.3.53149.log.gz 2023-12-10T23:40:10.104 INFO:teuthology.orchestra.run.smithi076.stderr: 88.7% -- replaced with /var/log/ceph/cluster1-client.mirror.44001.log.gz 2023-12-10T23:40:10.104 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55921.log 2023-12-10T23:40:10.105 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.57714.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55808.log 2023-12-10T23:40:10.105 INFO:teuthology.orchestra.run.smithi076.stderr: 87.7%/var/log/ceph/cluster2-client.mirror.55921.log: -- replaced with /var/log/ceph/cluster1-client.mirror.57714.log.gz 2023-12-10T23:40:10.106 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57668.log 2023-12-10T23:40:10.106 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.55808.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.55808.log.gz 2023-12-10T23:40:10.107 INFO:teuthology.orchestra.run.smithi076.stderr: 92.6%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56939.log 2023-12-10T23:40:10.107 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.55921.log.gz 2023-12-10T23:40:10.107 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.57668.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47123.log 2023-12-10T23:40:10.108 INFO:teuthology.orchestra.run.smithi076.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57668.log.gz 2023-12-10T23:40:10.108 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.56939.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.56939.log.gz 2023-12-10T23:40:10.108 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55343.log 2023-12-10T23:40:10.109 INFO:teuthology.orchestra.run.smithi076.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.46426.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50191.log 2023-12-10T23:40:10.109 INFO:teuthology.orchestra.run.smithi076.stderr: 2023-12-10T23:40:10.110 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.47123.log: /var/log/ceph/cluster2-client.mirror.55343.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55343.log.gz 2023-12-10T23:40:10.113 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49664.log 2023-12-10T23:40:10.115 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54480.log 2023-12-10T23:40:10.115 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.50191.log: /var/log/ceph/cluster2-client.mirror.49664.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.47123.log.gz 2023-12-10T23:40:10.115 INFO:teuthology.orchestra.run.smithi076.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50191.log.gz 2023-12-10T23:40:10.115 INFO:teuthology.orchestra.run.smithi076.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49664.log.gz 2023-12-10T23:40:10.116 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53922.log 2023-12-10T23:40:10.116 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.54480.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70547.log 2023-12-10T23:40:10.116 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.53922.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67360.log 2023-12-10T23:40:10.116 INFO:teuthology.orchestra.run.smithi076.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.54480.log.gz 2023-12-10T23:40:10.117 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.70547.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.53922.log.gz 2023-12-10T23:40:10.117 INFO:teuthology.orchestra.run.smithi076.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70547.log.gz 2023-12-10T23:40:10.118 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69314.log 2023-12-10T23:40:10.118 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46671.log 2023-12-10T23:40:10.118 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.67360.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50955.log 2023-12-10T23:40:10.118 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.69314.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.69314.log.gz 2023-12-10T23:40:10.119 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.46671.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.67360.log.gz 2023-12-10T23:40:10.119 INFO:teuthology.orchestra.run.smithi076.stderr: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.46671.log.gz 2023-12-10T23:40:10.119 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48807.log 2023-12-10T23:40:10.119 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47533.log 2023-12-10T23:40:10.119 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.50955.log: /var/log/ceph/cluster2-client.mirror.48807.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40835.log 2023-12-10T23:40:10.120 INFO:teuthology.orchestra.run.smithi076.stderr: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.50955.log.gz 2023-12-10T23:40:10.120 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.47533.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77921.log 2023-12-10T23:40:10.120 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.40835.log: 58.0% -- replaced with /var/log/ceph/cluster2-client.mirror.40835.log.gz 2023-12-10T23:40:10.121 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74476.log 2023-12-10T23:40:10.121 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.77921.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.77921.log.gz 2023-12-10T23:40:10.121 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39911.log 2023-12-10T23:40:10.123 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.74476.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.74476.log.gz 2023-12-10T23:40:10.123 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68342.log 2023-12-10T23:40:10.124 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.39911.log: 45.9% -- replaced with /var/log/ceph/cluster2-client.mirror.39911.log.gz 2023-12-10T23:40:10.125 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54074.log 2023-12-10T23:40:10.125 INFO:teuthology.orchestra.run.smithi076.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47533.log.gz 2023-12-10T23:40:10.125 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.68342.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.68342.log.gz 2023-12-10T23:40:10.126 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75642.log 2023-12-10T23:40:10.126 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49316.log 2023-12-10T23:40:10.127 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.54074.log: /var/log/ceph/cluster2-client.mirror.75642.log: 80.0% -- replaced with /var/log/ceph/cluster2-client.mirror.75642.log.gz 2023-12-10T23:40:10.127 INFO:teuthology.orchestra.run.smithi076.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54074.log.gz 2023-12-10T23:40:10.127 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57804.log 2023-12-10T23:40:10.128 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63953.log 2023-12-10T23:40:10.129 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.49316.log: /var/log/ceph/cluster2-client.mirror.57804.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57804.log.gz 2023-12-10T23:40:10.142 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62483.log 2023-12-10T23:40:10.142 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.63953.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.63953.log.gz 2023-12-10T23:40:10.143 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63102.log 2023-12-10T23:40:10.144 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.62483.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.62483.log.gz 2023-12-10T23:40:10.144 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52977.log 2023-12-10T23:40:10.147 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.63102.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63102.log.gz 2023-12-10T23:40:10.147 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55366.log 2023-12-10T23:40:10.148 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.52977.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52977.log.gz 2023-12-10T23:40:10.148 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42402.log 2023-12-10T23:40:10.148 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.55366.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.55366.log.gz 2023-12-10T23:40:10.149 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64554.log 2023-12-10T23:40:10.149 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.42402.log: 74.1% -- replaced with /var/log/ceph/cluster2-client.mirror.42402.log.gz 2023-12-10T23:40:10.150 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62946.log 2023-12-10T23:40:10.151 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.64554.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.64554.log.gz 2023-12-10T23:40:10.151 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56140.log 2023-12-10T23:40:10.152 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.62946.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.62946.log.gz 2023-12-10T23:40:10.153 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67444.log 2023-12-10T23:40:10.154 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.56140.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.56140.log.gz 2023-12-10T23:40:10.154 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53787.log 2023-12-10T23:40:10.155 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.67444.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.67444.log.gz 2023-12-10T23:40:10.155 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70120.log 2023-12-10T23:40:10.157 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.53787.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.53787.log.gz 2023-12-10T23:40:10.157 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67816.log 2023-12-10T23:40:10.158 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.70120.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70120.log.gz 2023-12-10T23:40:10.158 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43958.log 2023-12-10T23:40:10.159 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.67816.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.67816.log.gz 2023-12-10T23:40:10.159 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55963.log 2023-12-10T23:40:10.161 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.43958.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43958.log.gz 2023-12-10T23:40:10.161 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52520.log 2023-12-10T23:40:10.162 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.55963.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55963.log.gz 2023-12-10T23:40:10.162 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64532.log 2023-12-10T23:40:10.163 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.52520.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52520.log.gz 2023-12-10T23:40:10.163 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64887.log 2023-12-10T23:40:10.166 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.64532.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.64532.log.gz 2023-12-10T23:40:10.166 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78931.log 2023-12-10T23:40:10.167 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.64887.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.64887.log.gz 2023-12-10T23:40:10.167 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77418.log 2023-12-10T23:40:10.168 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.admin.78931.log: 7.1% 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51893.log.gz 2023-12-10T23:40:10.168 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.78931.log.gz 2023-12-10T23:40:10.168 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76135.log 2023-12-10T23:40:10.171 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.77418.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70007.log 2023-12-10T23:40:10.171 INFO:teuthology.orchestra.run.smithi076.stderr: 86.7%/var/log/ceph/cluster2-client.mirror.76135.log: -- replaced with /var/log/ceph/cluster1-client.mirror.77418.log.gz 2023-12-10T23:40:10.171 INFO:teuthology.orchestra.run.smithi076.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.76135.log.gz 2023-12-10T23:40:10.171 INFO:teuthology.orchestra.run.smithi076.stderr: 93.8%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70142.log 2023-12-10T23:40:10.171 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.50042.log.gz 2023-12-10T23:40:10.171 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.70007.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40985.log 2023-12-10T23:40:10.171 INFO:teuthology.orchestra.run.smithi076.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70007.log.gz/var/log/ceph/cluster1-client.mirror.70142.log: 2023-12-10T23:40:10.171 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42493.log 2023-12-10T23:40:10.171 INFO:teuthology.orchestra.run.smithi076.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70142.log.gz 2023-12-10T23:40:10.172 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.40985.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51093.log 2023-12-10T23:40:10.173 INFO:teuthology.orchestra.run.smithi076.stderr: 87.9%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77167.log 2023-12-10T23:40:10.173 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.40985.log.gz 2023-12-10T23:40:10.173 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.42493.log: /var/log/ceph/cluster2-client.mirror.51093.log: 73.9% -- replaced with /var/log/ceph/cluster1-client.mirror.42493.log.gz 2023-12-10T23:40:10.173 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57074.log 2023-12-10T23:40:10.173 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.77167.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76276.log 2023-12-10T23:40:10.174 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.57074.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.57074.log.gz 2023-12-10T23:40:10.174 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59659.log 2023-12-10T23:40:10.174 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.76276.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.76276.log.gz 2023-12-10T23:40:10.174 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69805.log 2023-12-10T23:40:10.175 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.59659.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.59659.log.gz 2023-12-10T23:40:10.175 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50000.log 2023-12-10T23:40:10.176 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.69805.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69805.log.gz 2023-12-10T23:40:10.176 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56319.log 2023-12-10T23:40:10.177 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.50000.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50000.log.gz 2023-12-10T23:40:10.177 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63279.log 2023-12-10T23:40:10.178 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.56319.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56319.log.gz 2023-12-10T23:40:10.178 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51362.log 2023-12-10T23:40:10.179 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.63279.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.63279.log.gz 2023-12-10T23:40:10.179 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69715.log 2023-12-10T23:40:10.180 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.51362.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.51362.log.gz 2023-12-10T23:40:10.180 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75560.log 2023-12-10T23:40:10.181 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.69715.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69715.log.gz 2023-12-10T23:40:10.181 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74743.log 2023-12-10T23:40:10.182 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.75560.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.75560.log.gz 2023-12-10T23:40:10.182 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48429.log 2023-12-10T23:40:10.183 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.74743.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.74743.log.gz 2023-12-10T23:40:10.183 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45070.log 2023-12-10T23:40:10.184 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.48429.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48429.log.gz 2023-12-10T23:40:10.184 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75893.log 2023-12-10T23:40:10.190 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.45070.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45070.log.gz 2023-12-10T23:40:10.190 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61001.log 2023-12-10T23:40:10.191 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.75893.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.75893.log.gz 2023-12-10T23:40:10.191 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52757.log 2023-12-10T23:40:10.192 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.61001.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61001.log.gz 2023-12-10T23:40:10.192 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43432.log 2023-12-10T23:40:10.193 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.52757.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52757.log.gz 2023-12-10T23:40:10.194 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73582.log 2023-12-10T23:40:10.195 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.43432.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43432.log.gz 2023-12-10T23:40:10.195 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52836.log 2023-12-10T23:40:10.196 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.73582.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.73582.log.gz 2023-12-10T23:40:10.196 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.34962.log 2023-12-10T23:40:10.197 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.52836.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52836.log.gz 2023-12-10T23:40:10.198 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76732.log 2023-12-10T23:40:10.198 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.admin.34962.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.34962.log.gz 2023-12-10T23:40:10.198 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39345.log 2023-12-10T23:40:10.199 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.76732.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.76732.log.gz 2023-12-10T23:40:10.199 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73562.log 2023-12-10T23:40:10.200 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.admin.39345.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39345.log.gz 2023-12-10T23:40:10.200 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50698.log 2023-12-10T23:40:10.201 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.73562.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.73562.log.gz 2023-12-10T23:40:10.201 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39107.log 2023-12-10T23:40:10.217 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.50698.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46250.log 2023-12-10T23:40:10.218 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.admin.39107.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39107.log.gz 2023-12-10T23:40:10.218 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44544.log 2023-12-10T23:40:10.219 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.46250.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46250.log.gz 2023-12-10T23:40:10.233 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65100.log 2023-12-10T23:40:10.239 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.44544.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44544.log.gz 2023-12-10T23:40:10.253 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55164.log 2023-12-10T23:40:10.254 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.65100.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65100.log.gz 2023-12-10T23:40:10.262 INFO:teuthology.orchestra.run.smithi033.stderr: 91.4% -- replaced with /var/log/ceph/cluster2-mgr.x.log.gz 2023-12-10T23:40:10.269 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70052.log 2023-12-10T23:40:10.270 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.55164.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.55164.log.gz 2023-12-10T23:40:10.272 INFO:teuthology.orchestra.run.smithi076.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.76157.log.gz 2023-12-10T23:40:10.272 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58031.log 2023-12-10T23:40:10.273 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.70052.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70052.log.gz 2023-12-10T23:40:10.274 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55231.log 2023-12-10T23:40:10.275 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.58031.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58031.log.gz 2023-12-10T23:40:10.275 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46628.log 2023-12-10T23:40:10.277 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.55231.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.55231.log.gz 2023-12-10T23:40:10.278 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51986.log 2023-12-10T23:40:10.279 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.46628.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46628.log.gz 2023-12-10T23:40:10.279 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42871.log 2023-12-10T23:40:10.280 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.51986.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51986.log.gz 2023-12-10T23:40:10.280 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57985.log 2023-12-10T23:40:10.281 INFO:teuthology.orchestra.run.smithi076.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49419.log.gz 2023-12-10T23:40:10.294 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.42871.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47103.log 2023-12-10T23:40:10.294 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.57985.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57985.log.gz 2023-12-10T23:40:10.295 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76882.log 2023-12-10T23:40:10.301 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.47103.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47103.log.gz 2023-12-10T23:40:10.301 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.1.log 2023-12-10T23:40:10.302 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.76882.log: 87.1% -- replaced with /var/log/ceph/cluster2-client.mirror.76882.log.gz 2023-12-10T23:40:10.302 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47060.log 2023-12-10T23:40:10.309 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-osd.1.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54097.log 2023-12-10T23:40:10.316 INFO:teuthology.orchestra.run.smithi033.stderr: 91.3% -- replaced with /var/log/ceph/cluster1-mgr.x.log.gz 2023-12-10T23:40:10.325 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.47060.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47060.log.gz 2023-12-10T23:40:10.329 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67381.log 2023-12-10T23:40:10.330 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.54097.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54097.log.gz 2023-12-10T23:40:10.349 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78473.log 2023-12-10T23:40:10.350 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.67381.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.67381.log.gz 2023-12-10T23:40:10.358 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51399.log 2023-12-10T23:40:10.358 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.admin.78473.log: 9.2% -- replaced with /var/log/ceph/cluster2-client.admin.78473.log.gz 2023-12-10T23:40:10.366 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60195.log 2023-12-10T23:40:10.366 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.51399.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51399.log.gz 2023-12-10T23:40:10.368 INFO:teuthology.orchestra.run.smithi076.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.75113.log.gz 2023-12-10T23:40:10.368 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52023.log 2023-12-10T23:40:10.370 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.60195.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.60195.log.gz 2023-12-10T23:40:10.371 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55273.log 2023-12-10T23:40:10.371 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.52023.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52023.log.gz 2023-12-10T23:40:10.372 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39600.log 2023-12-10T23:40:10.372 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.55273.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55273.log.gz 2023-12-10T23:40:10.373 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48889.log 2023-12-10T23:40:10.373 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.admin.39600.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39600.log.gz 2023-12-10T23:40:10.374 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52777.log 2023-12-10T23:40:10.381 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.48889.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67215.log 2023-12-10T23:40:10.382 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.52777.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52777.log.gz 2023-12-10T23:40:10.389 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76570.log 2023-12-10T23:40:10.391 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.67215.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.67215.log.gz 2023-12-10T23:40:10.406 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78496.log 2023-12-10T23:40:10.406 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.76570.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.76570.log.gz 2023-12-10T23:40:10.426 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75831.log 2023-12-10T23:40:10.426 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.78496.log: 10.2% -- replaced with /var/log/ceph/cluster2-client.mirror.78496.log.gz 2023-12-10T23:40:10.434 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64307.log 2023-12-10T23:40:10.434 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.75831.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.75831.log.gz 2023-12-10T23:40:10.442 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70300.log 2023-12-10T23:40:10.443 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.64307.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.64307.log.gz 2023-12-10T23:40:10.449 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48275.log 2023-12-10T23:40:10.450 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.70300.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.70300.log.gz 2023-12-10T23:40:10.456 INFO:teuthology.orchestra.run.smithi076.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49316.log.gz 2023-12-10T23:40:10.456 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43066.log 2023-12-10T23:40:10.457 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.48275.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48275.log.gz 2023-12-10T23:40:10.458 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44980.log 2023-12-10T23:40:10.458 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.43066.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.43066.log.gz 2023-12-10T23:40:10.459 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73623.log 2023-12-10T23:40:10.459 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.44980.log: 88.7% -- replaced with /var/log/ceph/cluster1-client.mirror.44980.log.gz 2023-12-10T23:40:10.460 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61222.log 2023-12-10T23:40:10.460 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.73623.log: 87.1% -- replaced with /var/log/ceph/cluster2-client.mirror.73623.log.gz 2023-12-10T23:40:10.461 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50735.log 2023-12-10T23:40:10.462 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.61222.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61222.log.gz 2023-12-10T23:40:10.462 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74992.log 2023-12-10T23:40:10.463 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.50735.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50735.log.gz 2023-12-10T23:40:10.464 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68466.log 2023-12-10T23:40:10.465 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.74992.log: 87.1% -- replaced with /var/log/ceph/cluster2-client.mirror.74992.log.gz 2023-12-10T23:40:10.465 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62068.log 2023-12-10T23:40:10.466 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.68466.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.68466.log.gz 2023-12-10T23:40:10.466 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43020.log 2023-12-10T23:40:10.467 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.62068.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.62068.log.gz 2023-12-10T23:40:10.468 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77904.log 2023-12-10T23:40:10.469 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.43020.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.43020.log.gz 2023-12-10T23:40:10.469 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69625.log 2023-12-10T23:40:10.469 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.77904.log: 67.9% -- replaced with /var/log/ceph/cluster1-client.mirror.77904.log.gz 2023-12-10T23:40:10.470 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75305.log 2023-12-10T23:40:10.470 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.69625.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.69625.log.gz 2023-12-10T23:40:10.471 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65473.log 2023-12-10T23:40:10.471 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.75305.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.75305.log.gz 2023-12-10T23:40:10.472 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78773.log 2023-12-10T23:40:10.474 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.65473.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.65473.log.gz 2023-12-10T23:40:10.474 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47696.log 2023-12-10T23:40:10.475 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.78773.log: 6.1% -- replaced with /var/log/ceph/cluster2-client.mirror.78773.log.gz 2023-12-10T23:40:10.475 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42356.log 2023-12-10T23:40:10.476 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.47696.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.47696.log.gz 2023-12-10T23:40:10.476 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75471.log 2023-12-10T23:40:10.477 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.42356.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.42356.log.gz 2023-12-10T23:40:10.477 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49775.log 2023-12-10T23:40:10.478 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.75471.log: 83.5% -- replaced with /var/log/ceph/cluster1-client.mirror.75471.log.gz 2023-12-10T23:40:10.478 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62109.log 2023-12-10T23:40:10.479 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.49775.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49775.log.gz 2023-12-10T23:40:10.479 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-mon.a.log 2023-12-10T23:40:10.481 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.62109.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.62109.log.gz 2023-12-10T23:40:10.482 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70502.log 2023-12-10T23:40:10.498 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-mon.a.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57443.log 2023-12-10T23:40:10.498 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.70502.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70502.log.gz 2023-12-10T23:40:10.507 INFO:teuthology.orchestra.run.smithi076.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.77167.log.gz 2023-12-10T23:40:10.510 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69917.log 2023-12-10T23:40:10.511 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.57443.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57443.log.gz 2023-12-10T23:40:10.511 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45924.log 2023-12-10T23:40:10.511 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.69917.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69917.log.gz 2023-12-10T23:40:10.512 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55696.log 2023-12-10T23:40:10.512 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.45924.log: 89.2% -- replaced with /var/log/ceph/cluster1-client.mirror.45924.log.gz 2023-12-10T23:40:10.513 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74972.log 2023-12-10T23:40:10.515 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.55696.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55696.log.gz 2023-12-10T23:40:10.515 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47338.log 2023-12-10T23:40:10.516 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.74972.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.74972.log.gz 2023-12-10T23:40:10.517 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58365.log 2023-12-10T23:40:10.517 INFO:teuthology.orchestra.run.smithi076.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48807.log.gz 2023-12-10T23:40:10.522 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.47338.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.47338.log.gz 2023-12-10T23:40:10.522 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69850.log 2023-12-10T23:40:10.523 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.58365.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58673.log 2023-12-10T23:40:10.523 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.69850.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.58365.log.gz 2023-12-10T23:40:10.523 INFO:teuthology.orchestra.run.smithi076.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.69850.log.gz 2023-12-10T23:40:10.525 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42926.log 2023-12-10T23:40:10.525 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.58673.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53406.log 2023-12-10T23:40:10.525 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.42926.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.58673.log.gz 2023-12-10T23:40:10.525 INFO:teuthology.orchestra.run.smithi076.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42926.log.gz 2023-12-10T23:40:10.525 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65743.log 2023-12-10T23:40:10.526 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.53406.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65146.log 2023-12-10T23:40:10.526 INFO:teuthology.orchestra.run.smithi076.stderr: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.53406.log.gz 2023-12-10T23:40:10.526 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.65743.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65743.log.gz 2023-12-10T23:40:10.527 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40688.log 2023-12-10T23:40:10.528 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54051.log 2023-12-10T23:40:10.528 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.65146.log: /var/log/ceph/cluster2-client.mirror.40688.log: 55.8% -- replaced with /var/log/ceph/cluster2-client.mirror.40688.log.gz 2023-12-10T23:40:10.528 INFO:teuthology.orchestra.run.smithi076.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.65146.log.gz 2023-12-10T23:40:10.528 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61956.log 2023-12-10T23:40:10.529 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58988.log 2023-12-10T23:40:10.529 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.54051.log: /var/log/ceph/cluster2-client.mirror.61956.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.54051.log.gz 2023-12-10T23:40:10.530 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66036.log 2023-12-10T23:40:10.530 INFO:teuthology.orchestra.run.smithi076.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.61956.log.gz 2023-12-10T23:40:10.531 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.58988.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58988.log.gz 2023-12-10T23:40:10.531 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75013.log 2023-12-10T23:40:10.532 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62900.log 2023-12-10T23:40:10.533 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.75013.log: /var/log/ceph/cluster2-client.mirror.66036.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.75013.log.gz 2023-12-10T23:40:10.533 INFO:teuthology.orchestra.run.smithi076.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.66036.log.gz 2023-12-10T23:40:10.534 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69422.log 2023-12-10T23:40:10.534 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65588.log 2023-12-10T23:40:10.535 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.62900.log: /var/log/ceph/cluster1-client.mirror.69422.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.69422.log.gz 2023-12-10T23:40:10.535 INFO:teuthology.orchestra.run.smithi076.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.62900.log.gz 2023-12-10T23:40:10.535 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59096.log 2023-12-10T23:40:10.537 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68792.log 2023-12-10T23:40:10.537 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.65588.log: 93.8%/var/log/ceph/cluster2-client.mirror.59096.log: -- replaced with /var/log/ceph/cluster2-client.mirror.51093.log.gz 2023-12-10T23:40:10.537 INFO:teuthology.orchestra.run.smithi076.stderr: 87.5% 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59096.log.gz 2023-12-10T23:40:10.537 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.65588.log.gz 2023-12-10T23:40:10.537 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39210.log 2023-12-10T23:40:10.537 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.68792.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.76186.log 2023-12-10T23:40:10.539 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.admin.39210.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76611.log 2023-12-10T23:40:10.539 INFO:teuthology.orchestra.run.smithi076.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.68792.log.gz 2023-12-10T23:40:10.539 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39210.log.gz 2023-12-10T23:40:10.540 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47936.log 2023-12-10T23:40:10.540 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.2.76186.log: /var/log/ceph/cluster2-client.mirror.76611.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71664.log 2023-12-10T23:40:10.540 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.47936.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.76611.log.gz 2023-12-10T23:40:10.540 INFO:teuthology.orchestra.run.smithi076.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47936.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69580.log 2023-12-10T23:40:10.540 INFO:teuthology.orchestra.run.smithi076.stderr: 2023-12-10T23:40:10.540 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.71664.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.71664.log.gz 2023-12-10T23:40:10.540 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48255.log 2023-12-10T23:40:10.542 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74005.log 2023-12-10T23:40:10.542 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.69580.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.69580.log.gz 2023-12-10T23:40:10.542 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.48255.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63144.log 2023-12-10T23:40:10.542 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.74005.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48255.log.gz 2023-12-10T23:40:10.542 INFO:teuthology.orchestra.run.smithi076.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.74005.log.gz 2023-12-10T23:40:10.542 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70615.log 2023-12-10T23:40:10.542 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66471.log 2023-12-10T23:40:10.544 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.63144.log: /var/log/ceph/cluster1-client.mirror.70615.log: 86.9% 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.63144.log.gz 2023-12-10T23:40:10.544 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.70615.log.gz 2023-12-10T23:40:10.544 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55521.log 2023-12-10T23:40:10.544 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.66471.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60322.log 2023-12-10T23:40:10.544 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.55521.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55521.log.gz 2023-12-10T23:40:10.545 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63694.log 2023-12-10T23:40:10.545 INFO:teuthology.orchestra.run.smithi076.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.66471.log.gz 2023-12-10T23:40:10.545 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.60322.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.60322.log.gz 2023-12-10T23:40:10.545 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67671.log 2023-12-10T23:40:10.546 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.63694.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63694.log.gz 2023-12-10T23:40:10.546 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69692.log 2023-12-10T23:40:10.547 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43241.log 2023-12-10T23:40:10.547 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.67671.log: /var/log/ceph/cluster1-client.mirror.69692.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69692.log.gz 2023-12-10T23:40:10.548 INFO:teuthology.orchestra.run.smithi076.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.67671.log.gz 2023-12-10T23:40:10.548 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.34938.log 2023-12-10T23:40:10.549 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47896.log 2023-12-10T23:40:10.549 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.43241.log: /var/log/ceph/cluster2-client.admin.34938.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.34938.log.gz 2023-12-10T23:40:10.549 INFO:teuthology.orchestra.run.smithi076.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43241.log.gz 2023-12-10T23:40:10.549 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.40006.log 2023-12-10T23:40:10.550 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40477.log 2023-12-10T23:40:10.550 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.47896.log: /var/log/ceph/cluster2-client.mirror.1.40006.log: 73.6% -- replaced with /var/log/ceph/cluster2-client.mirror.1.40006.log.gz 2023-12-10T23:40:10.550 INFO:teuthology.orchestra.run.smithi076.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47896.log.gz 2023-12-10T23:40:10.551 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70187.log 2023-12-10T23:40:10.551 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65000.log 2023-12-10T23:40:10.551 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.admin.40477.log: /var/log/ceph/cluster1-client.mirror.70187.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.admin.40477.log.gz 2023-12-10T23:40:10.552 INFO:teuthology.orchestra.run.smithi076.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70187.log.gz 2023-12-10T23:40:10.552 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56605.log 2023-12-10T23:40:10.554 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.65000.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.65000.log.gz 2023-12-10T23:40:10.554 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47017.log 2023-12-10T23:40:10.554 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46180.log 2023-12-10T23:40:10.555 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.56605.log: /var/log/ceph/cluster2-client.mirror.47017.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56605.log.gz 2023-12-10T23:40:10.555 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58244.log 2023-12-10T23:40:10.555 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.46180.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46180.log.gz 2023-12-10T23:40:10.556 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42192.log 2023-12-10T23:40:10.557 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.58244.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.58244.log.gz 2023-12-10T23:40:10.557 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43615.log 2023-12-10T23:40:10.558 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.42192.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.42192.log.gz 2023-12-10T23:40:10.558 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48607.log 2023-12-10T23:40:10.559 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.43615.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.43615.log.gz 2023-12-10T23:40:10.559 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60059.log 2023-12-10T23:40:10.560 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.48607.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47017.log.gz 2023-12-10T23:40:10.560 INFO:teuthology.orchestra.run.smithi076.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48607.log.gz 2023-12-10T23:40:10.561 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60979.log 2023-12-10T23:40:10.561 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50334.log 2023-12-10T23:40:10.562 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.60059.log: /var/log/ceph/cluster1-client.mirror.60979.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.60059.log.gz 2023-12-10T23:40:10.562 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52003.log 2023-12-10T23:40:10.563 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.50334.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60979.log.gz 2023-12-10T23:40:10.572 INFO:teuthology.orchestra.run.smithi076.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.2.76186.log.gz 2023-12-10T23:40:10.577 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.77887.log 2023-12-10T23:40:10.578 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39270.log 2023-12-10T23:40:10.578 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.52003.log: /var/log/ceph/cluster1-client.admin.77887.log: 70.8% -- replaced with /var/log/ceph/cluster1-client.admin.77887.log.gz 2023-12-10T23:40:10.579 INFO:teuthology.orchestra.run.smithi076.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52003.log.gz 2023-12-10T23:40:10.579 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42236.log 2023-12-10T23:40:10.579 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67712.log 2023-12-10T23:40:10.580 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.admin.39270.log: /var/log/ceph/cluster2-client.mirror.42236.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39270.log.gz 2023-12-10T23:40:10.580 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65657.log 2023-12-10T23:40:10.581 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.67712.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.67712.log.gz 2023-12-10T23:40:10.582 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71402.log 2023-12-10T23:40:10.582 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.65657.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.65657.log.gz 2023-12-10T23:40:10.583 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64329.log 2023-12-10T23:40:10.583 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.71402.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.71402.log.gz 2023-12-10T23:40:10.584 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43263.log 2023-12-10T23:40:10.585 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.64329.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64329.log.gz 2023-12-10T23:40:10.585 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57760.log 2023-12-10T23:40:10.585 INFO:teuthology.orchestra.run.smithi076.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.42236.log.gz 2023-12-10T23:40:10.593 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.43263.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51458.log 2023-12-10T23:40:10.594 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.57760.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.57760.log.gz 2023-12-10T23:40:10.595 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71087.log 2023-12-10T23:40:10.596 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.51458.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51458.log.gz 2023-12-10T23:40:10.596 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77533.log 2023-12-10T23:40:10.597 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.71087.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.71087.log.gz 2023-12-10T23:40:10.597 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44157.log 2023-12-10T23:40:10.598 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.77533.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.77533.log.gz 2023-12-10T23:40:10.598 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57423.log 2023-12-10T23:40:10.604 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.44157.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44157.log.gz 2023-12-10T23:40:10.604 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49096.log 2023-12-10T23:40:10.605 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.57423.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.57423.log.gz 2023-12-10T23:40:10.606 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60391.log 2023-12-10T23:40:10.607 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.49096.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49096.log.gz 2023-12-10T23:40:10.607 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54120.log 2023-12-10T23:40:10.608 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.60391.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.60391.log.gz 2023-12-10T23:40:10.608 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68639.log 2023-12-10T23:40:10.609 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.54120.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.54120.log.gz 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50698.log.gz 2023-12-10T23:40:10.610 INFO:teuthology.orchestra.run.smithi076.stderr: 2023-12-10T23:40:10.610 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55738.log 2023-12-10T23:40:10.610 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.68639.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47997.log 2023-12-10T23:40:10.611 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.55738.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55738.log.gz 2023-12-10T23:40:10.612 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52740.log 2023-12-10T23:40:10.612 INFO:teuthology.orchestra.run.smithi076.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.68639.log.gz 2023-12-10T23:40:10.612 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.47997.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.77695.log 2023-12-10T23:40:10.613 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.52740.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.52740.log.gz 2023-12-10T23:40:10.613 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47318.log 2023-12-10T23:40:10.614 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.admin.77695.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47997.log.gz 2023-12-10T23:40:10.614 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.77695.log.gz 2023-12-10T23:40:10.614 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52600.log 2023-12-10T23:40:10.615 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46121.log 2023-12-10T23:40:10.615 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.47318.log: /var/log/ceph/cluster2-client.mirror.52600.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52600.log.gz 2023-12-10T23:40:10.615 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47776.log 2023-12-10T23:40:10.619 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.46121.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47318.log.gz 2023-12-10T23:40:10.620 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71723.log 2023-12-10T23:40:10.621 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.47776.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.47776.log.gz 2023-12-10T23:40:10.621 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40818.log 2023-12-10T23:40:10.637 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.71723.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41928.log 2023-12-10T23:40:10.638 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.40818.log: 58.3% -- replaced with /var/log/ceph/cluster2-client.mirror.40818.log.gz 2023-12-10T23:40:10.638 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53640.log 2023-12-10T23:40:10.639 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.admin.41928.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.41928.log.gz 2023-12-10T23:40:10.649 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48217.log 2023-12-10T23:40:10.650 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.53640.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.53640.log.gz 2023-12-10T23:40:10.651 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42827.log 2023-12-10T23:40:10.673 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.48217.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61180.log 2023-12-10T23:40:10.681 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.42827.log: 93.5%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59030.log 2023-12-10T23:40:10.682 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.42827.log.gz 2023-12-10T23:40:10.683 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.61180.log: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.61180.log.gz 2023-12-10T23:40:10.691 INFO:teuthology.orchestra.run.smithi076.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.42871.log.gz 2023-12-10T23:40:10.691 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52067.log 2023-12-10T23:40:10.692 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.59030.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59030.log.gz 2023-12-10T23:40:10.693 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73939.log 2023-12-10T23:40:10.694 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.52067.log: 94.0% -- replaced with /var/log/ceph/cluster1-client.mirror.52067.log.gz 2023-12-10T23:40:10.694 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43480.log 2023-12-10T23:40:10.695 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.73939.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.73939.log.gz 2023-12-10T23:40:10.695 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47977.log 2023-12-10T23:40:10.696 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.43480.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43480.log.gz 2023-12-10T23:40:10.697 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66533.log 2023-12-10T23:40:10.698 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.47977.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.47977.log.gz 2023-12-10T23:40:10.698 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71335.log 2023-12-10T23:40:10.700 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.66533.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.66533.log.gz 2023-12-10T23:40:10.700 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54943.log 2023-12-10T23:40:10.701 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.71335.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.71335.log.gz 2023-12-10T23:40:10.701 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77372.log 2023-12-10T23:40:10.702 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.54943.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.54943.log.gz 2023-12-10T23:40:10.702 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44345.log 2023-12-10T23:40:10.703 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.77372.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.77372.log.gz 2023-12-10T23:40:10.703 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50312.log 2023-12-10T23:40:10.704 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.44345.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44345.log.gz 2023-12-10T23:40:10.704 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42034.log 2023-12-10T23:40:10.705 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.50312.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50312.log.gz 2023-12-10T23:40:10.705 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69445.log 2023-12-10T23:40:10.706 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.42034.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.42034.log.gz 2023-12-10T23:40:10.707 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52314.log 2023-12-10T23:40:10.708 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.69445.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.69445.log.gz 2023-12-10T23:40:10.708 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69895.log 2023-12-10T23:40:10.708 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.52314.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52314.log.gz 2023-12-10T23:40:10.709 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75722.log 2023-12-10T23:40:10.710 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.69895.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.69895.log.gz 2023-12-10T23:40:10.710 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56342.log 2023-12-10T23:40:10.710 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.75722.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.75722.log.gz 2023-12-10T23:40:10.711 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50656.log 2023-12-10T23:40:10.712 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.56342.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.56342.log.gz 2023-12-10T23:40:10.712 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74242.log 2023-12-10T23:40:10.713 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.50656.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50656.log.gz 2023-12-10T23:40:10.713 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77510.log 2023-12-10T23:40:10.714 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.74242.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.74242.log.gz 2023-12-10T23:40:10.715 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44608.log 2023-12-10T23:40:10.715 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.77510.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.77510.log.gz 2023-12-10T23:40:10.716 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57962.log 2023-12-10T23:40:10.716 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.44608.log: 73.7% -- replaced with /var/log/ceph/cluster2-client.mirror.44608.log.gz 2023-12-10T23:40:10.717 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76964.log 2023-12-10T23:40:10.717 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.57962.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.57962.log.gz 2023-12-10T23:40:10.718 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72669.log 2023-12-10T23:40:10.718 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.76964.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.76964.log.gz 2023-12-10T23:40:10.719 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74283.log 2023-12-10T23:40:10.720 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.72669.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.72669.log.gz 2023-12-10T23:40:10.720 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54806.log 2023-12-10T23:40:10.721 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.74283.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.74283.log.gz 2023-12-10T23:40:10.721 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54829.log 2023-12-10T23:40:10.722 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.54806.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54806.log.gz 2023-12-10T23:40:10.722 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65995.log 2023-12-10T23:40:10.723 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.54829.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.54829.log.gz 2023-12-10T23:40:10.723 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78293.log 2023-12-10T23:40:10.724 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.65995.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.65995.log.gz 2023-12-10T23:40:10.724 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45545.log 2023-12-10T23:40:10.725 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.78293.log: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.78293.log.gz 2023-12-10T23:40:10.725 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50972.log 2023-12-10T23:40:10.726 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.45545.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.45545.log.gz 2023-12-10T23:40:10.726 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74221.log 2023-12-10T23:40:10.729 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.50972.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50972.log.gz 2023-12-10T23:40:10.730 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45720.log 2023-12-10T23:40:10.730 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.74221.log: 87.1% -- replaced with /var/log/ceph/cluster2-client.mirror.74221.log.gz 2023-12-10T23:40:10.730 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45901.log 2023-12-10T23:40:10.731 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.45720.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45720.log.gz 2023-12-10T23:40:10.731 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67836.log 2023-12-10T23:40:10.732 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.45901.log: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.45901.log.gz 2023-12-10T23:40:10.732 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48517.log 2023-12-10T23:40:10.734 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.67836.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.67836.log.gz 2023-12-10T23:40:10.735 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64151.log 2023-12-10T23:40:10.735 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.48517.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48517.log.gz 2023-12-10T23:40:10.736 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39751.log 2023-12-10T23:40:10.736 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.64151.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.64151.log.gz 2023-12-10T23:40:10.736 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39574.log 2023-12-10T23:40:10.737 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.admin.39751.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39751.log.gz 2023-12-10T23:40:10.737 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51221.log 2023-12-10T23:40:10.738 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.admin.39574.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39574.log.gz 2023-12-10T23:40:10.739 INFO:teuthology.orchestra.run.smithi076.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48889.log.gz 2023-12-10T23:40:10.739 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78031.log 2023-12-10T23:40:10.740 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.51221.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51221.log.gz 2023-12-10T23:40:10.740 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72804.log 2023-12-10T23:40:10.741 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.78031.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78031.log.gz 2023-12-10T23:40:10.741 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71290.log 2023-12-10T23:40:10.742 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.72804.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.72804.log.gz 2023-12-10T23:40:10.742 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49116.log 2023-12-10T23:40:10.743 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.71290.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71290.log.gz 2023-12-10T23:40:10.743 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79044.log 2023-12-10T23:40:10.744 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.49116.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49116.log.gz 2023-12-10T23:40:10.744 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43767.log 2023-12-10T23:40:10.745 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.admin.79044.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79044.log.gz 2023-12-10T23:40:10.745 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60711.log 2023-12-10T23:40:10.746 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.admin.43767.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.43767.log.gz 2023-12-10T23:40:10.747 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70165.log 2023-12-10T23:40:10.749 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.60711.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60711.log.gz 2023-12-10T23:40:10.749 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78807.log 2023-12-10T23:40:10.750 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.70165.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.70165.log.gz 2023-12-10T23:40:10.750 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69267.log 2023-12-10T23:40:10.751 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.78807.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.78807.log.gz 2023-12-10T23:40:10.751 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61707.log 2023-12-10T23:40:10.754 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.69267.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.69267.log.gz 2023-12-10T23:40:10.754 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.34997.log 2023-12-10T23:40:10.755 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.61707.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.61707.log.gz 2023-12-10T23:40:10.755 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53830.log 2023-12-10T23:40:10.756 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.admin.34997.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.34997.log.gz 2023-12-10T23:40:10.756 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78796.log 2023-12-10T23:40:10.757 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.53830.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53830.log.gz 2023-12-10T23:40:10.757 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64468.log 2023-12-10T23:40:10.758 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.78796.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.78796.log.gz 2023-12-10T23:40:10.758 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59846.log 2023-12-10T23:40:10.759 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.64468.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.64468.log.gz 2023-12-10T23:40:10.760 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64491.log 2023-12-10T23:40:10.761 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.59846.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.59846.log.gz 2023-12-10T23:40:10.761 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52678.log 2023-12-10T23:40:10.762 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.64491.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.64491.log.gz 2023-12-10T23:40:10.762 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43546.log 2023-12-10T23:40:10.763 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.52678.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52678.log.gz 2023-12-10T23:40:10.764 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72928.log 2023-12-10T23:40:10.765 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.43546.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.43546.log.gz 2023-12-10T23:40:10.765 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47656.log 2023-12-10T23:40:10.766 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.72928.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.72928.log.gz 2023-12-10T23:40:10.766 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54244.log 2023-12-10T23:40:10.767 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.47656.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.47656.log.gz 2023-12-10T23:40:10.768 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58719.log 2023-12-10T23:40:10.769 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.54244.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.54244.log.gz 2023-12-10T23:40:10.769 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66327.log 2023-12-10T23:40:10.770 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.58719.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.58719.log.gz 2023-12-10T23:40:10.770 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52193.log 2023-12-10T23:40:10.772 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.66327.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.66327.log.gz 2023-12-10T23:40:10.772 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76985.log 2023-12-10T23:40:10.773 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.52193.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52193.log.gz 2023-12-10T23:40:10.773 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57846.log 2023-12-10T23:40:10.774 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.76985.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.76985.log.gz 2023-12-10T23:40:10.774 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66079.log 2023-12-10T23:40:10.775 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.57846.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57846.log.gz 2023-12-10T23:40:10.776 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56273.log 2023-12-10T23:40:10.776 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.66079.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.66079.log.gz 2023-12-10T23:40:10.777 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68598.log 2023-12-10T23:40:10.778 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.56273.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56273.log.gz 2023-12-10T23:40:10.778 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49167.log 2023-12-10T23:40:10.779 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.68598.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.68598.log.gz 2023-12-10T23:40:10.779 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46270.log 2023-12-10T23:40:10.780 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.49167.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.49167.log.gz 2023-12-10T23:40:10.780 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70592.log 2023-12-10T23:40:10.781 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.46270.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46270.log.gz 2023-12-10T23:40:10.782 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61598.log 2023-12-10T23:40:10.782 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.70592.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70592.log.gz 2023-12-10T23:40:10.783 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48067.log 2023-12-10T23:40:10.784 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.61598.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61598.log.gz 2023-12-10T23:40:10.784 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.61752.log 2023-12-10T23:40:10.785 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.48067.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48067.log.gz 2023-12-10T23:40:10.785 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60915.log 2023-12-10T23:40:10.791 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.0.61752.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.71723.log.gz 2023-12-10T23:40:10.791 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44221.log 2023-12-10T23:40:10.792 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.60915.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60915.log.gz 2023-12-10T23:40:10.793 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62547.log 2023-12-10T23:40:10.801 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.44221.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48988.log 2023-12-10T23:40:10.809 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.62547.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49257.log 2023-12-10T23:40:10.811 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.48988.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62547.log.gz 2023-12-10T23:40:10.817 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49514.log 2023-12-10T23:40:10.818 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.49257.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.49257.log.gz 2023-12-10T23:40:10.824 INFO:teuthology.orchestra.run.smithi076.stderr: 92.9% -- replaced with /var/log/ceph/cluster1-client.mirror.0.61752.log.gz 2023-12-10T23:40:10.824 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64910.log 2023-12-10T23:40:10.825 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.49514.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49514.log.gz 2023-12-10T23:40:10.825 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50131.log 2023-12-10T23:40:10.826 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.64910.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.64910.log.gz 2023-12-10T23:40:10.826 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59442.log 2023-12-10T23:40:10.835 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.50131.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63887.log 2023-12-10T23:40:10.838 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.59442.log: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.59442.log.gz 2023-12-10T23:40:10.841 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65905.log 2023-12-10T23:40:10.842 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.63887.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.63887.log.gz 2023-12-10T23:40:10.857 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54400.log 2023-12-10T23:40:10.858 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.65905.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.65905.log.gz 2023-12-10T23:40:10.865 INFO:teuthology.orchestra.run.smithi076.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50334.log.gz 2023-12-10T23:40:10.873 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58584.log 2023-12-10T23:40:10.875 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.54400.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54400.log.gz 2023-12-10T23:40:10.876 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69737.log 2023-12-10T23:40:10.877 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.58584.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.58584.log.gz 2023-12-10T23:40:10.890 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68854.log 2023-12-10T23:40:10.890 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.69737.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.69737.log.gz 2023-12-10T23:40:10.898 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39425.log 2023-12-10T23:40:10.898 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.68854.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.68854.log.gz 2023-12-10T23:40:10.905 INFO:teuthology.orchestra.run.smithi076.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.43263.log.gz 2023-12-10T23:40:10.905 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77395.log 2023-12-10T23:40:10.906 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.admin.39425.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39425.log.gz 2023-12-10T23:40:10.906 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62173.log 2023-12-10T23:40:10.907 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.77395.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.77395.log.gz 2023-12-10T23:40:10.908 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48480.log 2023-12-10T23:40:10.909 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.62173.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.62173.log.gz 2023-12-10T23:40:10.909 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56365.log 2023-12-10T23:40:10.910 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.48480.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48480.log.gz 2023-12-10T23:40:10.910 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65317.log 2023-12-10T23:40:10.911 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.56365.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.56365.log.gz 2023-12-10T23:40:10.911 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47209.log 2023-12-10T23:40:10.912 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.65317.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65317.log.gz 2023-12-10T23:40:10.913 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.77671.log 2023-12-10T23:40:10.919 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.47209.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.47209.log.gz 2023-12-10T23:40:10.919 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78451.log 2023-12-10T23:40:10.919 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.admin.77671.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.77671.log.gz 2023-12-10T23:40:10.920 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49862.log 2023-12-10T23:40:10.921 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.78451.log: 9.2% -- replaced with /var/log/ceph/cluster2-client.mirror.78451.log.gz 2023-12-10T23:40:10.921 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72948.log 2023-12-10T23:40:10.922 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.49862.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49862.log.gz 2023-12-10T23:40:10.922 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39503.log 2023-12-10T23:40:10.923 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.72948.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.72948.log.gz 2023-12-10T23:40:10.923 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45439.log 2023-12-10T23:40:10.924 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.admin.39503.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39503.log.gz 2023-12-10T23:40:10.925 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54546.log 2023-12-10T23:40:10.925 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.45439.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.45439.log.gz 2023-12-10T23:40:10.926 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62569.log 2023-12-10T23:40:10.927 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.54546.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54546.log.gz 2023-12-10T23:40:10.927 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62216.log 2023-12-10T23:40:10.928 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.62569.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.62569.log.gz 2023-12-10T23:40:10.929 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66823.log 2023-12-10T23:40:10.929 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.62216.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62216.log.gz 2023-12-10T23:40:10.930 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55029.log 2023-12-10T23:40:10.931 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.66823.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.66823.log.gz 2023-12-10T23:40:10.931 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67588.log 2023-12-10T23:40:10.932 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.55029.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55029.log.gz 2023-12-10T23:40:10.933 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78420.log 2023-12-10T23:40:10.934 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.67588.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.67588.log.gz 2023-12-10T23:40:10.935 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44245.log 2023-12-10T23:40:10.935 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.admin.78420.log: 7.1% -- replaced with /var/log/ceph/cluster2-client.admin.78420.log.gz 2023-12-10T23:40:10.936 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.45743.log 2023-12-10T23:40:10.937 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.admin.44245.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.44245.log.gz 2023-12-10T23:40:10.937 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61644.log 2023-12-10T23:40:10.940 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.0.45743.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.46121.log.gz 2023-12-10T23:40:10.949 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77650.log 2023-12-10T23:40:10.951 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.61644.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.61644.log.gz 2023-12-10T23:40:10.951 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78310.log 2023-12-10T23:40:10.952 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.77650.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.77650.log.gz 2023-12-10T23:40:10.952 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56626.log 2023-12-10T23:40:10.953 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.78310.log: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.78310.log.gz 2023-12-10T23:40:10.953 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53727.log 2023-12-10T23:40:10.954 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.56626.log: 93.0% -- replaced with /var/log/ceph/cluster1-client.mirror.0.45743.log.gz 2023-12-10T23:40:10.955 INFO:teuthology.orchestra.run.smithi076.stderr: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.56626.log.gz 2023-12-10T23:40:10.955 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72846.log 2023-12-10T23:40:10.956 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60082.log 2023-12-10T23:40:10.957 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.53727.log: /var/log/ceph/cluster2-client.mirror.72846.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.72846.log.gz 2023-12-10T23:40:10.958 INFO:teuthology.orchestra.run.smithi076.stderr: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.53727.log.gz 2023-12-10T23:40:10.958 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45947.log 2023-12-10T23:40:10.959 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62837.log 2023-12-10T23:40:10.959 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.60082.log: /var/log/ceph/cluster1-client.mirror.45947.log: 87.7% 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.45947.log.gz 2023-12-10T23:40:10.959 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.60082.log.gz 2023-12-10T23:40:10.960 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.43285.log 2023-12-10T23:40:10.960 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76903.log 2023-12-10T23:40:10.961 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.62837.log: /var/log/ceph/cluster2-client.mirror.3.43285.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.62837.log.gz 2023-12-10T23:40:10.961 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56851.log 2023-12-10T23:40:10.962 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.76903.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.76903.log.gz 2023-12-10T23:40:10.962 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61976.log 2023-12-10T23:40:10.965 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.56851.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.56851.log.gz 2023-12-10T23:40:10.965 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40391.log 2023-12-10T23:40:10.966 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.61976.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.61976.log.gz 2023-12-10T23:40:10.966 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78052.log 2023-12-10T23:40:10.967 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.40391.log: 53.8% -- replaced with /var/log/ceph/cluster1-client.mirror.40391.log.gz 2023-12-10T23:40:10.967 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67981.log 2023-12-10T23:40:10.968 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.78052.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78052.log.gz 2023-12-10T23:40:10.968 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54569.log 2023-12-10T23:40:10.969 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.67981.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.67981.log.gz 2023-12-10T23:40:10.970 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59401.log 2023-12-10T23:40:10.970 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.54569.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54569.log.gz 2023-12-10T23:40:10.971 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77279.log 2023-12-10T23:40:10.972 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.59401.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59401.log.gz 2023-12-10T23:40:10.972 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74326.log 2023-12-10T23:40:10.973 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.77279.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.77279.log.gz 2023-12-10T23:40:10.973 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68383.log 2023-12-10T23:40:10.974 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.74326.log: 77.0% -- replaced with /var/log/ceph/cluster1-client.mirror.74326.log.gz 2023-12-10T23:40:10.974 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41954.log 2023-12-10T23:40:10.976 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.68383.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.68383.log.gz 2023-12-10T23:40:10.976 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47424.log 2023-12-10T23:40:10.977 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.admin.41954.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.41954.log.gz 2023-12-10T23:40:10.977 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2.audit.log 2023-12-10T23:40:10.983 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.47424.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.47424.log.gz 2023-12-10T23:40:10.983 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42784.log 2023-12-10T23:40:10.987 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2.audit.log: 93.1% -- replaced with /var/log/ceph/cluster2.audit.log.gz 2023-12-10T23:40:10.987 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69872.log 2023-12-10T23:40:10.993 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.42784.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42784.log.gz 2023-12-10T23:40:10.993 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60555.log 2023-12-10T23:40:10.994 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.69872.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.69872.log.gz 2023-12-10T23:40:10.994 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51115.log 2023-12-10T23:40:10.995 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.60555.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.60555.log.gz 2023-12-10T23:40:10.996 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45503.log 2023-12-10T23:40:11.000 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.51115.log: 95.5% -- replaced with /var/log/ceph/cluster1-client.mirror.51115.log.gz 2023-12-10T23:40:11.000 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.68169.log 2023-12-10T23:40:11.006 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.45503.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45503.log.gz 2023-12-10T23:40:11.006 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66761.log 2023-12-10T23:40:11.014 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.2.68169.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56117.log 2023-12-10T23:40:11.017 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.66761.log: 87.8% 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.66761.log.gz 2023-12-10T23:40:11.017 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.2.68169.log.gz 2023-12-10T23:40:11.017 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49377.log 2023-12-10T23:40:11.018 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.56117.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.56117.log.gz 2023-12-10T23:40:11.018 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75071.log 2023-12-10T23:40:11.019 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.49377.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49377.log.gz 2023-12-10T23:40:11.020 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64956.log 2023-12-10T23:40:11.021 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.75071.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.75071.log.gz 2023-12-10T23:40:11.021 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52404.log 2023-12-10T23:40:11.022 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.64956.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.64956.log.gz 2023-12-10T23:40:11.023 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62677.log 2023-12-10T23:40:11.023 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.52404.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52404.log.gz 2023-12-10T23:40:11.024 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71357.log 2023-12-10T23:40:11.025 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.62677.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62677.log.gz 2023-12-10T23:40:11.025 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45204.log 2023-12-10T23:40:11.026 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.71357.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.71357.log.gz 2023-12-10T23:40:11.026 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43135.log 2023-12-10T23:40:11.027 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.admin.45204.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.45204.log.gz 2023-12-10T23:40:11.028 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63849.log 2023-12-10T23:40:11.029 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.43135.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.43135.log.gz 2023-12-10T23:40:11.029 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44777.log 2023-12-10T23:40:11.030 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.63849.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63849.log.gz 2023-12-10T23:40:11.030 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48748.log 2023-12-10T23:40:11.031 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.44777.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.44777.log.gz 2023-12-10T23:40:11.032 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2.log 2023-12-10T23:40:11.033 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.48748.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48748.log.gz 2023-12-10T23:40:11.033 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51500.log 2023-12-10T23:40:11.039 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2.log: 88.0% -- replaced with /var/log/ceph/cluster2.log.gz 2023-12-10T23:40:11.039 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71110.log 2023-12-10T23:40:11.054 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.51500.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41181.log 2023-12-10T23:40:11.054 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.71110.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71110.log.gz 2023-12-10T23:40:11.058 INFO:teuthology.orchestra.run.smithi033.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-mon.a.log.gz 2023-12-10T23:40:11.065 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67319.log 2023-12-10T23:40:11.073 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.41181.log: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.3.43285.log.gz 2023-12-10T23:40:11.081 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75263.log 2023-12-10T23:40:11.082 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.67319.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.67319.log.gz 2023-12-10T23:40:11.083 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39184.log 2023-12-10T23:40:11.083 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.75263.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.75263.log.gz 2023-12-10T23:40:11.098 INFO:teuthology.orchestra.run.smithi076.stderr:gzip 93.9% -5 -- replaced with /var/log/ceph/cluster2-client.mirror.44221.log.gz --verbose 2023-12-10T23:40:11.098 INFO:teuthology.orchestra.run.smithi076.stderr: -- /var/log/ceph/cluster1-client.mirror.65969.log 2023-12-10T23:40:11.098 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.admin.39184.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39184.log.gz 2023-12-10T23:40:11.099 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52173.log 2023-12-10T23:40:11.100 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.65969.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.65969.log.gz 2023-12-10T23:40:11.101 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40255.log 2023-12-10T23:40:11.102 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.52173.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52173.log.gz 2023-12-10T23:40:11.102 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77963.log 2023-12-10T23:40:11.103 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.admin.40255.log: 80.4% -- replaced with /var/log/ceph/cluster1-client.admin.40255.log.gz 2023-12-10T23:40:11.103 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.41066.log 2023-12-10T23:40:11.104 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.77963.log: 67.7% -- replaced with /var/log/ceph/cluster1-client.mirror.77963.log.gz 2023-12-10T23:40:11.104 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55583.log 2023-12-10T23:40:11.118 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.2.41066.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51014.log 2023-12-10T23:40:11.119 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.55583.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55583.log.gz 2023-12-10T23:40:11.119 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77441.log 2023-12-10T23:40:11.133 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.51014.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59768.log 2023-12-10T23:40:11.143 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.77441.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48217.log.gz 2023-12-10T23:40:11.145 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71681.log 2023-12-10T23:40:11.146 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.59768.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.59768.log.gz 2023-12-10T23:40:11.149 INFO:teuthology.orchestra.run.smithi076.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.77441.log.gz 2023-12-10T23:40:11.162 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74415.log 2023-12-10T23:40:11.162 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.71681.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.71681.log.gz 2023-12-10T23:40:11.166 INFO:teuthology.orchestra.run.smithi076.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48988.log.gz 2023-12-10T23:40:11.174 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.77997.log 2023-12-10T23:40:11.174 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.74415.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.74415.log.gz 2023-12-10T23:40:11.175 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.51522.log 2023-12-10T23:40:11.176 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.admin.77997.log: 70.4% -- replaced with /var/log/ceph/cluster1-client.admin.77997.log.gz 2023-12-10T23:40:11.176 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62306.log 2023-12-10T23:40:11.185 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.0.51522.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69143.log 2023-12-10T23:40:11.188 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.62306.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.62306.log.gz 2023-12-10T23:40:11.209 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60959.log 2023-12-10T23:40:11.211 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.69143.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.69143.log.gz 2023-12-10T23:40:11.218 INFO:teuthology.orchestra.run.smithi076.stderr: 93.3%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69670.log 2023-12-10T23:40:11.220 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.60959.log: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.0.51522.log.gz 2023-12-10T23:40:11.220 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.60959.log.gz 2023-12-10T23:40:11.221 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53899.log 2023-12-10T23:40:11.222 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.69670.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.69670.log.gz 2023-12-10T23:40:11.222 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75913.log 2023-12-10T23:40:11.223 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.53899.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.53899.log.gz 2023-12-10T23:40:11.223 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65453.log 2023-12-10T23:40:11.224 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.75913.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.75913.log.gz 2023-12-10T23:40:11.224 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45133.log 2023-12-10T23:40:11.226 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.65453.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65453.log.gz 2023-12-10T23:40:11.227 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56048.log 2023-12-10T23:40:11.233 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.45133.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.45133.log.gz 2023-12-10T23:40:11.233 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47836.log 2023-12-10T23:40:11.234 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.56048.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.56048.log.gz 2023-12-10T23:40:11.234 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.50462.log 2023-12-10T23:40:11.235 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.47836.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47836.log.gz 2023-12-10T23:40:11.235 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44070.log 2023-12-10T23:40:11.252 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.0.50462.log: 93.1%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63519.log 2023-12-10T23:40:11.253 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.44070.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.44070.log.gz 2023-12-10T23:40:11.253 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.0.50462.log.gz 2023-12-10T23:40:11.253 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56784.log 2023-12-10T23:40:11.254 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.63519.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63519.log.gz 2023-12-10T23:40:11.255 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53342.log 2023-12-10T23:40:11.255 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.56784.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.56784.log.gz 2023-12-10T23:40:11.256 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49941.log 2023-12-10T23:40:11.257 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.53342.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.53342.log.gz 2023-12-10T23:40:11.257 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70435.log 2023-12-10T23:40:11.258 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.49941.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49941.log.gz 2023-12-10T23:40:11.259 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78612.log 2023-12-10T23:40:11.259 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.70435.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70435.log.gz 2023-12-10T23:40:11.260 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64174.log 2023-12-10T23:40:11.261 INFO:teuthology.orchestra.run.smithi076.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50131.log.gz 2023-12-10T23:40:11.261 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.admin.78612.log: 9.2% -- replaced with /var/log/ceph/cluster2-client.admin.78612.log.gz 2023-12-10T23:40:11.261 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43285.log 2023-12-10T23:40:11.262 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58944.log 2023-12-10T23:40:11.262 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.64174.log: /var/log/ceph/cluster1-client.mirror.43285.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.64174.log.gz 2023-12-10T23:40:11.262 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63827.log 2023-12-10T23:40:11.263 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.58944.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.58944.log.gz 2023-12-10T23:40:11.263 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73705.log 2023-12-10T23:40:11.266 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.63827.log: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.63827.log.gz 2023-12-10T23:40:11.266 INFO:teuthology.orchestra.run.smithi076.stderr: 95.3%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61400.log 2023-12-10T23:40:11.266 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.43285.log.gz 2023-12-10T23:40:11.267 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.73705.log: 87.1% -- replaced with /var/log/ceph/cluster2-client.mirror.73705.log.gz 2023-12-10T23:40:11.267 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63583.log 2023-12-10T23:40:11.268 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50213.log 2023-12-10T23:40:11.268 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.61400.log: /var/log/ceph/cluster1-client.mirror.63583.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.61400.log.gz 2023-12-10T23:40:11.268 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76447.log 2023-12-10T23:40:11.269 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.50213.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63583.log.gz 2023-12-10T23:40:11.278 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40340.log 2023-12-10T23:40:11.278 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.76447.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.76447.log.gz 2023-12-10T23:40:11.279 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54356.log 2023-12-10T23:40:11.280 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.40340.log: 57.1% -- replaced with /var/log/ceph/cluster1-client.mirror.40340.log.gz 2023-12-10T23:40:11.280 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.77190.log 2023-12-10T23:40:11.281 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.54356.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54356.log.gz 2023-12-10T23:40:11.281 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71537.log 2023-12-10T23:40:11.297 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.1.77190.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62460.log 2023-12-10T23:40:11.298 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.71537.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.71537.log.gz 2023-12-10T23:40:11.310 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78706.log 2023-12-10T23:40:11.310 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.62460.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.62460.log.gz 2023-12-10T23:40:11.315 INFO:teuthology.orchestra.run.smithi076.stderr: 93.8% -- replaced with /var/log/ceph/cluster1-client.mirror.2.41066.log.gz 2023-12-10T23:40:11.316 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48392.log 2023-12-10T23:40:11.316 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.78706.log: 11.2% -- replaced with /var/log/ceph/cluster2-client.mirror.78706.log.gz 2023-12-10T23:40:11.316 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40289.log 2023-12-10T23:40:11.317 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.48392.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48392.log.gz 2023-12-10T23:40:11.317 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69226.log 2023-12-10T23:40:11.318 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.40289.log: 54.0% -- replaced with /var/log/ceph/cluster1-client.mirror.40289.log.gz 2023-12-10T23:40:11.318 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.78920.log 2023-12-10T23:40:11.318 INFO:teuthology.orchestra.run.smithi076.stderr: 95.4% -- replaced with /var/log/ceph/cluster1-client.mirror.1.77190.log.gz 2023-12-10T23:40:11.319 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.69226.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.69226.log.gz 2023-12-10T23:40:11.320 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58967.log 2023-12-10T23:40:11.321 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.39914.log 2023-12-10T23:40:11.321 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.admin.78920.log: /var/log/ceph/cluster1-client.mirror.58967.log: 7.1% -- replaced with /var/log/ceph/cluster1-client.admin.78920.log.gz 2023-12-10T23:40:11.321 INFO:teuthology.orchestra.run.smithi076.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.58967.log.gz 2023-12-10T23:40:11.321 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57691.log 2023-12-10T23:40:11.322 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60105.log 2023-12-10T23:40:11.322 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.3.39914.log: /var/log/ceph/cluster1-client.mirror.57691.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.57691.log.gz 2023-12-10T23:40:11.333 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46316.log 2023-12-10T23:40:11.334 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.60105.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60105.log.gz 2023-12-10T23:40:11.335 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77005.log 2023-12-10T23:40:11.336 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.46316.log: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.46316.log.gz 2023-12-10T23:40:11.336 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67754.log 2023-12-10T23:40:11.337 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.77005.log: 87.1% -- replaced with /var/log/ceph/cluster2-client.mirror.77005.log.gz 2023-12-10T23:40:11.337 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51619.log 2023-12-10T23:40:11.338 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.67754.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.67754.log.gz 2023-12-10T23:40:11.339 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52661.log 2023-12-10T23:40:11.340 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.51619.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51619.log.gz 2023-12-10T23:40:11.340 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69293.log 2023-12-10T23:40:11.341 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.52661.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52661.log.gz 2023-12-10T23:40:11.341 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55435.log 2023-12-10T23:40:11.342 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.69293.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.69293.log.gz 2023-12-10T23:40:11.342 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68937.log 2023-12-10T23:40:11.343 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.55435.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.55435.log.gz 2023-12-10T23:40:11.343 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59602.log 2023-12-10T23:40:11.344 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.68937.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.68937.log.gz 2023-12-10T23:40:11.345 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76711.log 2023-12-10T23:40:11.345 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.59602.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59602.log.gz 2023-12-10T23:40:11.346 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65859.log 2023-12-10T23:40:11.361 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.76711.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66947.log 2023-12-10T23:40:11.362 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.65859.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.65859.log.gz 2023-12-10T23:40:11.367 INFO:teuthology.orchestra.run.smithi076.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.3.39914.log.gz 2023-12-10T23:40:11.376 INFO:teuthology.orchestra.run.smithi076.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51500.log.gz 2023-12-10T23:40:11.377 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70772.log 2023-12-10T23:40:11.377 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.66947.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56409.log 2023-12-10T23:40:11.378 INFO:teuthology.orchestra.run.smithi076.stderr: 87.9%/var/log/ceph/cluster1-client.mirror.70772.log: -- replaced with /var/log/ceph/cluster2-client.mirror.66947.log.gz 2023-12-10T23:40:11.379 INFO:teuthology.orchestra.run.smithi076.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.70772.log.gz 2023-12-10T23:40:11.379 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67277.log 2023-12-10T23:40:11.379 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42697.log 2023-12-10T23:40:11.381 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.56409.log: /var/log/ceph/cluster2-client.mirror.67277.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.67277.log.gz 2023-12-10T23:40:11.381 INFO:teuthology.orchestra.run.smithi076.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56409.log.gz 2023-12-10T23:40:11.381 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67071.log 2023-12-10T23:40:11.382 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40801.log 2023-12-10T23:40:11.382 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.42697.log: /var/log/ceph/cluster2-client.mirror.67071.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42697.log.gz 2023-12-10T23:40:11.382 INFO:teuthology.orchestra.run.smithi076.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.67071.log.gz 2023-12-10T23:40:11.382 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68300.log 2023-12-10T23:40:11.383 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54783.log 2023-12-10T23:40:11.383 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.40801.log: /var/log/ceph/cluster2-client.mirror.68300.log: 4.7% -- replaced with /var/log/ceph/cluster2-client.mirror.40801.log.gz 2023-12-10T23:40:11.384 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65680.log 2023-12-10T23:40:11.384 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.54783.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54783.log.gz 2023-12-10T23:40:11.385 INFO:teuthology.orchestra.run.smithi076.stderr: 90.1%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57005.log 2023-12-10T23:40:11.385 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.68300.log.gz 2023-12-10T23:40:11.385 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.65680.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.65680.log.gz 2023-12-10T23:40:11.385 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71629.log 2023-12-10T23:40:11.386 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70637.log 2023-12-10T23:40:11.387 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.57005.log: /var/log/ceph/cluster1-client.mirror.71629.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.71629.log.gz 2023-12-10T23:40:11.387 INFO:teuthology.orchestra.run.smithi076.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.57005.log.gz 2023-12-10T23:40:11.387 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61441.log 2023-12-10T23:40:11.388 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78760.log 2023-12-10T23:40:11.388 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.70637.log: /var/log/ceph/cluster1-client.mirror.61441.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.70637.log.gz 2023-12-10T23:40:11.388 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42300.log 2023-12-10T23:40:11.389 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.78760.log: 9.2% -- replaced with /var/log/ceph/cluster2-client.mirror.78760.log.gz 2023-12-10T23:40:11.389 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68508.log 2023-12-10T23:40:11.389 INFO:teuthology.orchestra.run.smithi076.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61441.log.gz 2023-12-10T23:40:11.406 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.42300.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57175.log 2023-12-10T23:40:11.407 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.68508.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68508.log.gz 2023-12-10T23:40:11.408 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71582.log 2023-12-10T23:40:11.409 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.57175.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.57175.log.gz 2023-12-10T23:40:11.409 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76549.log 2023-12-10T23:40:11.410 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.71582.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.71582.log.gz 2023-12-10T23:40:11.410 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56071.log 2023-12-10T23:40:11.411 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.76549.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.76549.log.gz 2023-12-10T23:40:11.412 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78183.log 2023-12-10T23:40:11.412 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.56071.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56071.log.gz 2023-12-10T23:40:11.413 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76652.log 2023-12-10T23:40:11.413 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.78183.log: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.78183.log.gz 2023-12-10T23:40:11.414 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.42402.log 2023-12-10T23:40:11.415 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.76652.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.76652.log.gz 2023-12-10T23:40:11.415 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39236.log 2023-12-10T23:40:11.418 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.0.42402.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51014.log.gz 2023-12-10T23:40:11.418 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68002.log 2023-12-10T23:40:11.419 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.admin.39236.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39236.log.gz 2023-12-10T23:40:11.420 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46974.log 2023-12-10T23:40:11.420 INFO:teuthology.orchestra.run.smithi076.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.0.42402.log.gz 2023-12-10T23:40:11.420 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.68002.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51034.log 2023-12-10T23:40:11.421 INFO:teuthology.orchestra.run.smithi076.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.68002.log.gz 2023-12-10T23:40:11.425 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.46974.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46974.log.gz 2023-12-10T23:40:11.426 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68486.log 2023-12-10T23:40:11.426 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45003.log 2023-12-10T23:40:11.426 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.51034.log: /var/log/ceph/cluster2-client.mirror.68486.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.51034.log.gz 2023-12-10T23:40:11.427 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78634.log 2023-12-10T23:40:11.428 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.45003.log: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.45003.log.gz 2023-12-10T23:40:11.429 INFO:teuthology.orchestra.run.smithi076.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.68486.log.gz 2023-12-10T23:40:11.429 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61575.log 2023-12-10T23:40:11.429 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.78634.log: 9.2% -- replaced with /var/log/ceph/cluster2-client.mirror.78634.log.gz 2023-12-10T23:40:11.430 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72907.log 2023-12-10T23:40:11.430 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63907.log 2023-12-10T23:40:11.430 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.61575.log: /var/log/ceph/cluster2-client.mirror.72907.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.61575.log.gz 2023-12-10T23:40:11.430 INFO:teuthology.orchestra.run.smithi076.stderr: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.72907.log.gz 2023-12-10T23:40:11.431 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66492.log 2023-12-10T23:40:11.431 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.63907.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.63907.log.gz 2023-12-10T23:40:11.432 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68225.log 2023-12-10T23:40:11.432 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51755.log 2023-12-10T23:40:11.433 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.66492.log: /var/log/ceph/cluster2-client.mirror.68225.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.66492.log.gz 2023-12-10T23:40:11.434 INFO:teuthology.orchestra.run.smithi076.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.68225.log.gz 2023-12-10T23:40:11.434 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56513.log 2023-12-10T23:40:11.434 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50797.log 2023-12-10T23:40:11.434 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.51755.log: /var/log/ceph/cluster2-client.mirror.56513.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.51755.log.gz 2023-12-10T23:40:11.434 INFO:teuthology.orchestra.run.smithi076.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.56513.log.gz 2023-12-10T23:40:11.434 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65042.log 2023-12-10T23:40:11.436 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44457.log 2023-12-10T23:40:11.436 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.50797.log: /var/log/ceph/cluster2-client.mirror.65042.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50797.log.gz 2023-12-10T23:40:11.436 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43679.log 2023-12-10T23:40:11.436 INFO:teuthology.orchestra.run.smithi076.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65042.log.gz 2023-12-10T23:40:11.436 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.44457.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.41181.log.gz 2023-12-10T23:40:11.436 INFO:teuthology.orchestra.run.smithi076.stderr: 88.7% -- replaced with /var/log/ceph/cluster1-client.mirror.44457.log.gz 2023-12-10T23:40:11.436 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43656.log 2023-12-10T23:40:11.437 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.43679.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67339.log 2023-12-10T23:40:11.437 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.43656.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47756.log 2023-12-10T23:40:11.438 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.67339.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.67339.log.gz 2023-12-10T23:40:11.438 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55251.log 2023-12-10T23:40:11.439 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.47756.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.47756.log.gz 2023-12-10T23:40:11.440 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62611.log 2023-12-10T23:40:11.441 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.55251.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43679.log.gz 2023-12-10T23:40:11.445 INFO:teuthology.orchestra.run.smithi076.stderr: 92.0%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78730.log 2023-12-10T23:40:11.445 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.55251.log.gz 2023-12-10T23:40:11.445 INFO:teuthology.orchestra.run.smithi076.stderr: 93.4%/var/log/ceph/cluster2-client.mirror.62611.log: -- replaced with /var/log/ceph/cluster1-client.mirror.43656.log.gz 2023-12-10T23:40:11.445 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77325.log 2023-12-10T23:40:11.445 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65634.log 2023-12-10T23:40:11.445 INFO:teuthology.orchestra.run.smithi076.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.62611.log.gz 2023-12-10T23:40:11.445 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.78730.log: /var/log/ceph/cluster1-client.mirror.77325.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58411.log 2023-12-10T23:40:11.445 INFO:teuthology.orchestra.run.smithi076.stderr: 9.2% -- replaced with /var/log/ceph/cluster2-client.mirror.78730.log.gz 2023-12-10T23:40:11.445 INFO:teuthology.orchestra.run.smithi076.stderr: 86.8%/var/log/ceph/cluster1-client.mirror.65634.log: -- replaced with /var/log/ceph/cluster1-client.mirror.77325.log.gz 2023-12-10T23:40:11.446 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74435.log 2023-12-10T23:40:11.446 INFO:teuthology.orchestra.run.smithi076.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.65634.log.gz 2023-12-10T23:40:11.446 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44365.log 2023-12-10T23:40:11.446 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.58411.log: /var/log/ceph/cluster2-client.mirror.74435.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59979.log 2023-12-10T23:40:11.446 INFO:teuthology.orchestra.run.smithi076.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58411.log.gz 2023-12-10T23:40:11.446 INFO:teuthology.orchestra.run.smithi076.stderr: 86.4%/var/log/ceph/cluster2-client.mirror.44365.log: -- replaced with /var/log/ceph/cluster2-client.mirror.74435.log.gz 2023-12-10T23:40:11.446 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39801.log 2023-12-10T23:40:11.451 INFO:teuthology.orchestra.run.smithi076.stderr: 88.5%/var/log/ceph/cluster1-client.mirror.59979.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78843.log 2023-12-10T23:40:11.451 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.44365.log.gz 2023-12-10T23:40:11.451 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.admin.39801.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78620.log 2023-12-10T23:40:11.452 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39801.log.gz/var/log/ceph/cluster2-client.mirror.78843.log: 2023-12-10T23:40:11.452 INFO:teuthology.orchestra.run.smithi076.stderr: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.78843.log.gz 2023-12-10T23:40:11.452 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53749.log 2023-12-10T23:40:11.452 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43176.log 2023-12-10T23:40:11.452 INFO:teuthology.orchestra.run.smithi076.stderr: 92.2%/var/log/ceph/cluster2-client.mirror.78620.log: -- replaced with /var/log/ceph/cluster1-client.mirror.59979.log.gz 2023-12-10T23:40:11.452 INFO:teuthology.orchestra.run.smithi076.stderr: 10.2% -- replaced with /var/log/ceph/cluster2-client.mirror.78620.log.gz 2023-12-10T23:40:11.452 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.53749.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57310.log 2023-12-10T23:40:11.452 INFO:teuthology.orchestra.run.smithi076.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53749.log.gz 2023-12-10T23:40:11.453 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53807.log 2023-12-10T23:40:11.453 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.43176.log: /var/log/ceph/cluster1-client.mirror.57310.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77579.log 2023-12-10T23:40:11.453 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.53807.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.57310.log.gz 2023-12-10T23:40:11.453 INFO:teuthology.orchestra.run.smithi076.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.53807.log.gz 2023-12-10T23:40:11.453 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61308.log 2023-12-10T23:40:11.453 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.77579.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39651.log 2023-12-10T23:40:11.453 INFO:teuthology.orchestra.run.smithi076.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.77579.log.gz 2023-12-10T23:40:11.453 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.61308.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52542.log 2023-12-10T23:40:11.453 INFO:teuthology.orchestra.run.smithi076.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.61308.log.gz 2023-12-10T23:40:11.453 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.admin.39651.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39651.log.gz 2023-12-10T23:40:11.454 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56253.log 2023-12-10T23:40:11.454 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.52542.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56646.log 2023-12-10T23:40:11.455 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.56253.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.56253.log.gz 2023-12-10T23:40:11.455 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48946.log 2023-12-10T23:40:11.455 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.56646.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.43176.log.gz 2023-12-10T23:40:11.455 INFO:teuthology.orchestra.run.smithi076.stderr: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56646.log.gz 2023-12-10T23:40:11.455 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70322.log 2023-12-10T23:40:11.456 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70705.log 2023-12-10T23:40:11.456 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.48946.log: /var/log/ceph/cluster1-client.mirror.70322.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.70322.log.gz 2023-12-10T23:40:11.457 INFO:teuthology.orchestra.run.smithi076.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48946.log.gz 2023-12-10T23:40:11.457 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76923.log 2023-12-10T23:40:11.458 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69040.log 2023-12-10T23:40:11.458 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.70705.log: /var/log/ceph/cluster2-client.mirror.76923.log: 87.1% -- replaced with /var/log/ceph/cluster1-client.mirror.70705.log.gz 2023-12-10T23:40:11.458 INFO:teuthology.orchestra.run.smithi076.stderr: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.76923.log.gz 2023-12-10T23:40:11.459 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69164.log 2023-12-10T23:40:11.459 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53963.log 2023-12-10T23:40:11.460 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.69040.log: /var/log/ceph/cluster2-client.mirror.69164.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.69040.log.gz 2023-12-10T23:40:11.460 INFO:teuthology.orchestra.run.smithi076.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.69164.log.gz 2023-12-10T23:40:11.460 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66637.log 2023-12-10T23:40:11.460 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.72691.log 2023-12-10T23:40:11.461 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.53963.log: /var/log/ceph/cluster2-client.mirror.66637.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.66637.log.gz 2023-12-10T23:40:11.461 INFO:teuthology.orchestra.run.smithi076.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.53963.log.gz 2023-12-10T23:40:11.461 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48084.log 2023-12-10T23:40:11.462 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.0.72691.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67650.log 2023-12-10T23:40:11.463 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.48084.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48084.log.gz 2023-12-10T23:40:11.463 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50777.log 2023-12-10T23:40:11.465 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.67650.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.67650.log.gz 2023-12-10T23:40:11.465 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75621.log 2023-12-10T23:40:11.481 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.50777.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65701.log 2023-12-10T23:40:11.483 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.75621.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.75621.log.gz 2023-12-10T23:40:11.486 INFO:teuthology.orchestra.run.smithi076.stderr: 92.6% -- replaced with /var/log/ceph/cluster1-client.mirror.0.72691.log.gz 2023-12-10T23:40:11.486 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60803.log 2023-12-10T23:40:11.488 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.65701.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65701.log.gz 2023-12-10T23:40:11.489 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76186.log 2023-12-10T23:40:11.490 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.60803.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.60803.log.gz 2023-12-10T23:40:11.490 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46951.log 2023-12-10T23:40:11.491 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.76186.log: 75.4% -- replaced with /var/log/ceph/cluster1-client.mirror.76186.log.gz 2023-12-10T23:40:11.491 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73726.log 2023-12-10T23:40:11.497 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.46951.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.46951.log.gz 2023-12-10T23:40:11.497 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59547.log 2023-12-10T23:40:11.498 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.73726.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.73726.log.gz 2023-12-10T23:40:11.499 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58501.log 2023-12-10T23:40:11.505 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.59547.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78739.log 2023-12-10T23:40:11.519 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.58501.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.58501.log.gz 2023-12-10T23:40:11.522 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58198.log 2023-12-10T23:40:11.522 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.78739.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.78739.log.gz 2023-12-10T23:40:11.529 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40948.log 2023-12-10T23:40:11.530 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.58198.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58198.log.gz 2023-12-10T23:40:11.539 INFO:teuthology.orchestra.run.smithi076.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59547.log.gz 2023-12-10T23:40:11.540 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73845.log 2023-12-10T23:40:11.541 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.40948.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.40948.log.gz 2023-12-10T23:40:11.541 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75216.log 2023-12-10T23:40:11.542 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.73845.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.73845.log.gz 2023-12-10T23:40:11.542 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41480.log 2023-12-10T23:40:11.546 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.75216.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.75216.log.gz 2023-12-10T23:40:11.546 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70345.log 2023-12-10T23:40:11.546 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.admin.41480.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.41480.log.gz 2023-12-10T23:40:11.547 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73476.log 2023-12-10T23:40:11.547 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.70345.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70345.log.gz 2023-12-10T23:40:11.547 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44114.log 2023-12-10T23:40:11.548 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.73476.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.73476.log.gz 2023-12-10T23:40:11.548 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58760.log 2023-12-10T23:40:11.551 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.44114.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.42300.log.gz 2023-12-10T23:40:11.553 INFO:teuthology.orchestra.run.smithi076.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44114.log.gz 2023-12-10T23:40:11.554 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51421.log 2023-12-10T23:40:11.554 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50089.log 2023-12-10T23:40:11.556 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.58760.log: /var/log/ceph/cluster2-client.mirror.51421.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58760.log.gz 2023-12-10T23:40:11.556 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67568.log 2023-12-10T23:40:11.557 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.50089.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50089.log.gz 2023-12-10T23:40:11.558 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63807.log 2023-12-10T23:40:11.558 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.67568.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.67568.log.gz 2023-12-10T23:40:11.559 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64620.log 2023-12-10T23:40:11.561 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.63807.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.63807.log.gz 2023-12-10T23:40:11.561 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42999.log 2023-12-10T23:40:11.561 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.64620.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.64620.log.gz 2023-12-10T23:40:11.562 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71425.log 2023-12-10T23:40:11.563 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.42999.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.42999.log.gz 2023-12-10T23:40:11.563 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61200.log 2023-12-10T23:40:11.564 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.71425.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.71425.log.gz 2023-12-10T23:40:11.564 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55941.log 2023-12-10T23:40:11.566 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.61200.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.61200.log.gz 2023-12-10T23:40:11.566 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68978.log 2023-12-10T23:40:11.568 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.55941.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55941.log.gz 2023-12-10T23:40:11.569 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73541.log 2023-12-10T23:40:11.569 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.68978.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.68978.log.gz 2023-12-10T23:40:11.570 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52483.log 2023-12-10T23:40:11.570 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.73541.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.73541.log.gz 2023-12-10T23:40:11.571 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66099.log 2023-12-10T23:40:11.572 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.52483.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.52483.log.gz 2023-12-10T23:40:11.572 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75366.log 2023-12-10T23:40:11.573 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.66099.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.66099.log.gz 2023-12-10T23:40:11.574 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42628.log 2023-12-10T23:40:11.574 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.75366.log: 87.1% -- replaced with /var/log/ceph/cluster2-client.mirror.75366.log.gz 2023-12-10T23:40:11.575 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68022.log 2023-12-10T23:40:11.575 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.42628.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.42628.log.gz 2023-12-10T23:40:11.576 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76016.log 2023-12-10T23:40:11.578 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.68022.log: 93.8% 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.68022.log.gz 2023-12-10T23:40:11.578 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.50213.log.gz 2023-12-10T23:40:11.578 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41066.log 2023-12-10T23:40:11.579 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39909.log 2023-12-10T23:40:11.579 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.76016.log: /var/log/ceph/cluster2-client.mirror.41066.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.76016.log.gz 2023-12-10T23:40:11.580 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61421.log 2023-12-10T23:40:11.580 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.39909.log: 79.1% -- replaced with /var/log/ceph/cluster2-client.mirror.39909.log.gz 2023-12-10T23:40:11.580 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69760.log 2023-12-10T23:40:11.582 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.61421.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61421.log.gz 2023-12-10T23:40:11.583 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62022.log 2023-12-10T23:40:11.583 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.69760.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.69760.log.gz 2023-12-10T23:40:11.584 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48847.log 2023-12-10T23:40:11.585 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.62022.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62022.log.gz 2023-12-10T23:40:11.585 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63190.log 2023-12-10T23:40:11.587 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.48847.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48847.log.gz 2023-12-10T23:40:11.587 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47467.log 2023-12-10T23:40:11.588 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.63190.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.63190.log.gz 2023-12-10T23:40:11.588 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42651.log 2023-12-10T23:40:11.594 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.47467.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.47467.log.gz 2023-12-10T23:40:11.594 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73162.log 2023-12-10T23:40:11.595 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.42651.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.42651.log.gz 2023-12-10T23:40:11.595 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50814.log 2023-12-10T23:40:11.597 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.73162.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.73162.log.gz 2023-12-10T23:40:11.598 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52463.log 2023-12-10T23:40:11.599 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.50814.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50814.log.gz 2023-12-10T23:40:11.599 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44957.log 2023-12-10T23:40:11.614 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.52463.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78344.log 2023-12-10T23:40:11.615 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.44957.log: 89.1% -- replaced with /var/log/ceph/cluster1-client.mirror.44957.log.gz 2023-12-10T23:40:11.628 INFO:teuthology.orchestra.run.smithi076.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52542.log.gz 2023-12-10T23:40:11.628 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58072.log 2023-12-10T23:40:11.629 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.78344.log: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.78344.log.gz 2023-12-10T23:40:11.629 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52698.log 2023-12-10T23:40:11.631 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.58072.log: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.58072.log.gz 2023-12-10T23:40:11.631 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43843.log 2023-12-10T23:40:11.632 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.52698.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52698.log.gz 2023-12-10T23:40:11.633 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61331.log 2023-12-10T23:40:11.638 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.43843.log: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.43843.log.gz 2023-12-10T23:40:11.638 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65813.log 2023-12-10T23:40:11.639 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.61331.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.61331.log.gz 2023-12-10T23:40:11.640 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69185.log 2023-12-10T23:40:11.640 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.65813.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.65813.log.gz 2023-12-10T23:40:11.641 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52580.log 2023-12-10T23:40:11.641 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.69185.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.69185.log.gz 2023-12-10T23:40:11.642 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78090.log 2023-12-10T23:40:11.643 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.52580.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52580.log.gz 2023-12-10T23:40:11.644 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62506.log 2023-12-10T23:40:11.644 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.78090.log: 57.4% -- replaced with /var/log/ceph/cluster1-client.mirror.78090.log.gz 2023-12-10T23:40:11.644 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62285.log 2023-12-10T23:40:11.645 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.62506.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.62506.log.gz 2023-12-10T23:40:11.645 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46204.log 2023-12-10T23:40:11.646 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.62285.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.62285.log.gz 2023-12-10T23:40:11.647 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57137.log 2023-12-10T23:40:11.647 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.46204.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46204.log.gz 2023-12-10T23:40:11.648 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69512.log 2023-12-10T23:40:11.649 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.57137.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57137.log.gz 2023-12-10T23:40:11.649 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75054.log 2023-12-10T23:40:11.650 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.69512.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.69512.log.gz 2023-12-10T23:40:11.651 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73455.log 2023-12-10T23:40:11.651 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.75054.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.75054.log.gz 2023-12-10T23:40:11.651 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51441.log 2023-12-10T23:40:11.652 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.73455.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.73455.log.gz 2023-12-10T23:40:11.653 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64798.log 2023-12-10T23:40:11.653 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.51441.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.51441.log.gz 2023-12-10T23:40:11.653 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51656.log 2023-12-10T23:40:11.654 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.64798.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64798.log.gz 2023-12-10T23:40:11.655 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49647.log 2023-12-10T23:40:11.655 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.51656.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51656.log.gz 2023-12-10T23:40:11.656 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66885.log 2023-12-10T23:40:11.657 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.49647.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.49647.log.gz 2023-12-10T23:40:11.657 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55412.log 2023-12-10T23:40:11.657 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.66885.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.66885.log.gz 2023-12-10T23:40:11.658 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70952.log 2023-12-10T23:40:11.659 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.55412.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.55412.log.gz 2023-12-10T23:40:11.659 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57351.log 2023-12-10T23:40:11.660 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.70952.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70952.log.gz 2023-12-10T23:40:11.660 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53036.log 2023-12-10T23:40:11.662 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.57351.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.57351.log.gz 2023-12-10T23:40:11.662 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75934.log 2023-12-10T23:40:11.677 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.53036.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49491.log 2023-12-10T23:40:11.678 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.75934.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.75934.log.gz 2023-12-10T23:40:11.690 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71200.log 2023-12-10T23:40:11.691 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.49491.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49491.log.gz 2023-12-10T23:40:11.707 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44630.log 2023-12-10T23:40:11.708 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.71200.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.71200.log.gz 2023-12-10T23:40:11.714 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68813.log 2023-12-10T23:40:11.722 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.44630.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52045.log 2023-12-10T23:40:11.722 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.68813.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.68813.log.gz 2023-12-10T23:40:11.729 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.68508.log 2023-12-10T23:40:11.737 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.52045.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73983.log 2023-12-10T23:40:11.745 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.0.68508.log: 91.7%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59892.log 2023-12-10T23:40:11.746 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.73983.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.73983.log.gz 2023-12-10T23:40:11.757 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.0.68508.log.gz 2023-12-10T23:40:11.761 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70075.log 2023-12-10T23:40:11.762 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.59892.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59892.log.gz 2023-12-10T23:40:11.773 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69400.log 2023-12-10T23:40:11.774 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.70075.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70075.log.gz 2023-12-10T23:40:11.790 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51283.log 2023-12-10T23:40:11.790 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.69400.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.69400.log.gz 2023-12-10T23:40:11.793 INFO:teuthology.orchestra.run.smithi076.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.41066.log.gz 2023-12-10T23:40:11.795 INFO:teuthology.orchestra.run.smithi076.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51421.log.gz 2023-12-10T23:40:11.795 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67526.log 2023-12-10T23:40:11.796 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.51283.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51283.log.gz 2023-12-10T23:40:11.797 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49825.log 2023-12-10T23:40:11.798 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.67526.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.67526.log.gz 2023-12-10T23:40:11.799 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45324.log 2023-12-10T23:40:11.799 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.49825.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.49825.log.gz 2023-12-10T23:40:11.800 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73220.log 2023-12-10T23:40:11.800 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.45324.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.45324.log.gz 2023-12-10T23:40:11.801 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50588.log 2023-12-10T23:40:11.802 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.73220.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.73220.log.gz 2023-12-10T23:40:11.802 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56982.log 2023-12-10T23:40:11.803 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.50588.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50588.log.gz 2023-12-10T23:40:11.804 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73183.log 2023-12-10T23:40:11.804 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.56982.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.56982.log.gz 2023-12-10T23:40:11.805 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55210.log 2023-12-10T23:40:11.806 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.73183.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.73183.log.gz 2023-12-10T23:40:11.806 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55877.log 2023-12-10T23:40:11.807 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.55210.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.55210.log.gz 2023-12-10T23:40:11.808 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78073.log 2023-12-10T23:40:11.808 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.55877.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55877.log.gz 2023-12-10T23:40:11.809 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48765.log 2023-12-10T23:40:11.809 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.78073.log: 10.2% -- replaced with /var/log/ceph/cluster1-client.mirror.78073.log.gz 2023-12-10T23:40:11.810 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64864.log 2023-12-10T23:40:11.811 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.48765.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48765.log.gz 2023-12-10T23:40:11.811 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54854.log 2023-12-10T23:40:11.812 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.64864.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50777.log.gz 2023-12-10T23:40:11.812 INFO:teuthology.orchestra.run.smithi076.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.64864.log.gz 2023-12-10T23:40:11.813 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77760.log 2023-12-10T23:40:11.814 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.54854.log: 95.5% -- replaced with /var/log/ceph/cluster1-client.mirror.54854.log.gz 2023-12-10T23:40:11.815 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73808.log 2023-12-10T23:40:11.815 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.77760.log: 55.8% -- replaced with /var/log/ceph/cluster1-client.mirror.77760.log.gz 2023-12-10T23:40:11.816 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60174.log 2023-12-10T23:40:11.816 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.73808.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.73808.log.gz 2023-12-10T23:40:11.817 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60690.log 2023-12-10T23:40:11.818 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.60174.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.60174.log.gz 2023-12-10T23:40:11.818 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45567.log 2023-12-10T23:40:11.819 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.60690.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.60690.log.gz 2023-12-10T23:40:11.819 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72887.log 2023-12-10T23:40:11.822 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.45567.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.76711.log.gz 2023-12-10T23:40:11.822 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57218.log 2023-12-10T23:40:11.823 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.72887.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.72887.log.gz 2023-12-10T23:40:11.823 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57115.log 2023-12-10T23:40:11.824 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.57218.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.57218.log.gz 2023-12-10T23:40:11.824 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70930.log 2023-12-10T23:40:11.826 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.57115.log: 91.8% -- replaced with /var/log/ceph/cluster1-client.mirror.57115.log.gz 2023-12-10T23:40:11.827 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47916.log 2023-12-10T23:40:11.827 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.70930.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.70930.log.gz 2023-12-10T23:40:11.828 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63081.log 2023-12-10T23:40:11.829 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.47916.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47916.log.gz 2023-12-10T23:40:11.829 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53853.log 2023-12-10T23:40:11.830 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.63081.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.63081.log.gz 2023-12-10T23:40:11.830 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52915.log 2023-12-10T23:40:11.831 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.53853.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.53853.log.gz 2023-12-10T23:40:11.831 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43378.log 2023-12-10T23:40:11.832 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.52915.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52915.log.gz 2023-12-10T23:40:11.833 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62527.log 2023-12-10T23:40:11.834 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.admin.43378.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.43378.log.gz 2023-12-10T23:40:11.834 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78683.log 2023-12-10T23:40:11.834 INFO:teuthology.orchestra.run.smithi076.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52463.log.gz 2023-12-10T23:40:11.836 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.62527.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.62527.log.gz 2023-12-10T23:40:11.837 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43721.log 2023-12-10T23:40:11.837 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.admin.78683.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68709.log 2023-12-10T23:40:11.837 INFO:teuthology.orchestra.run.smithi076.stderr: 10.2% -- replaced with /var/log/ceph/cluster2-client.admin.78683.log.gz 2023-12-10T23:40:11.838 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.43721.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62045.log 2023-12-10T23:40:11.838 INFO:teuthology.orchestra.run.smithi076.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.43721.log.gz 2023-12-10T23:40:11.839 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.68709.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.68709.log.gz 2023-12-10T23:40:11.839 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39426.log 2023-12-10T23:40:11.840 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55831.log 2023-12-10T23:40:11.840 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.62045.log: /var/log/ceph/cluster2-client.admin.39426.log: 0.0% 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.62045.log.gz 2023-12-10T23:40:11.840 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.39426.log.gz 2023-12-10T23:40:11.841 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58740.log 2023-12-10T23:40:11.841 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50639.log 2023-12-10T23:40:11.841 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.55831.log: /var/log/ceph/cluster2-client.mirror.58740.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.55831.log.gz 2023-12-10T23:40:11.843 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50292.log 2023-12-10T23:40:11.843 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.50639.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50639.log.gz 2023-12-10T23:40:11.843 INFO:teuthology.orchestra.run.smithi076.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58740.log.gz 2023-12-10T23:40:11.843 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61047.log 2023-12-10T23:40:11.844 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47404.log 2023-12-10T23:40:11.844 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.50292.log: /var/log/ceph/cluster2-client.mirror.61047.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50292.log.gz 2023-12-10T23:40:11.845 INFO:teuthology.orchestra.run.smithi076.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.61047.log.gz 2023-12-10T23:40:11.845 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69602.log 2023-12-10T23:40:11.845 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75872.log 2023-12-10T23:40:11.846 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.47404.log: /var/log/ceph/cluster1-client.mirror.69602.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69602.log.gz 2023-12-10T23:40:11.850 INFO:teuthology.orchestra.run.smithi076.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47404.log.gz 2023-12-10T23:40:11.850 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61529.log 2023-12-10T23:40:11.851 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70232.log 2023-12-10T23:40:11.852 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.75872.log: /var/log/ceph/cluster2-client.mirror.61529.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.75872.log.gz 2023-12-10T23:40:11.852 INFO:teuthology.orchestra.run.smithi076.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.61529.log.gz 2023-12-10T23:40:11.852 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54442.log 2023-12-10T23:40:11.853 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75852.log 2023-12-10T23:40:11.853 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.70232.log: /var/log/ceph/cluster2-client.mirror.54442.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70232.log.gz 2023-12-10T23:40:11.853 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50610.log 2023-12-10T23:40:11.853 INFO:teuthology.orchestra.run.smithi076.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54442.log.gz 2023-12-10T23:40:11.854 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.75852.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.75852.log.gz 2023-12-10T23:40:11.854 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73012.log 2023-12-10T23:40:11.855 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47736.log 2023-12-10T23:40:11.855 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.50610.log: /var/log/ceph/cluster1-client.mirror.73012.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.73012.log.gz 2023-12-10T23:40:11.869 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.54617.log 2023-12-10T23:40:11.871 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.47736.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.47736.log.gz 2023-12-10T23:40:11.871 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75810.log 2023-12-10T23:40:11.881 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.0.54617.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53571.log 2023-12-10T23:40:11.882 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.75810.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.75810.log.gz 2023-12-10T23:40:11.889 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45696.log 2023-12-10T23:40:11.890 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.53571.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53571.log.gz 2023-12-10T23:40:11.891 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45416.log 2023-12-10T23:40:11.901 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.45696.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.45696.log.gz 2023-12-10T23:40:11.905 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39418.log 2023-12-10T23:40:11.906 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.45416.log: 89.2% -- replaced with /var/log/ceph/cluster1-client.mirror.45416.log.gz 2023-12-10T23:40:11.921 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56536.log 2023-12-10T23:40:11.922 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.admin.39418.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39418.log.gz 2023-12-10T23:40:11.930 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47796.log 2023-12-10T23:40:11.930 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.56536.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.56536.log.gz 2023-12-10T23:40:11.936 INFO:teuthology.orchestra.run.smithi076.stderr: 93.6% -- replaced with /var/log/ceph/cluster1-client.mirror.0.54617.log.gz 2023-12-10T23:40:11.936 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59464.log 2023-12-10T23:40:11.937 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.47796.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.47796.log.gz 2023-12-10T23:40:11.938 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60215.log 2023-12-10T23:40:11.939 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.59464.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59464.log.gz 2023-12-10T23:40:11.939 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39969.log 2023-12-10T23:40:11.941 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.60215.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.60215.log.gz 2023-12-10T23:40:11.943 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61463.log 2023-12-10T23:40:11.943 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.39969.log: 81.5% -- replaced with /var/log/ceph/cluster1-client.mirror.39969.log.gz 2023-12-10T23:40:11.943 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70097.log 2023-12-10T23:40:11.944 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.61463.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61463.log.gz 2023-12-10T23:40:11.944 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42146.log 2023-12-10T23:40:11.945 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.70097.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70097.log.gz 2023-12-10T23:40:11.946 INFO:teuthology.orchestra.run.smithi076.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-mon.a.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66802.log 2023-12-10T23:40:11.946 INFO:teuthology.orchestra.run.smithi076.stderr: 2023-12-10T23:40:11.946 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.42146.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.42146.log.gz 2023-12-10T23:40:11.947 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52331.log 2023-12-10T23:40:11.947 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57939.log 2023-12-10T23:40:11.948 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.66802.log: /var/log/ceph/cluster2-client.mirror.52331.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.66802.log.gz 2023-12-10T23:40:11.948 INFO:teuthology.orchestra.run.smithi076.stderr: 88.7%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52235.log 2023-12-10T23:40:11.948 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.52331.log.gz 2023-12-10T23:40:11.949 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.57939.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.57939.log.gz 2023-12-10T23:40:11.949 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52898.log 2023-12-10T23:40:11.953 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70480.log 2023-12-10T23:40:11.954 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.52235.log: /var/log/ceph/cluster2-client.mirror.52898.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.52235.log.gz 2023-12-10T23:40:11.954 INFO:teuthology.orchestra.run.smithi076.stderr: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52898.log.gz 2023-12-10T23:40:11.954 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66347.log 2023-12-10T23:40:11.954 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47166.log 2023-12-10T23:40:11.954 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.70480.log: /var/log/ceph/cluster2-client.mirror.66347.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70480.log.gz 2023-12-10T23:40:11.954 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63496.log 2023-12-10T23:40:11.954 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.47166.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.66347.log.gz 2023-12-10T23:40:11.957 INFO:teuthology.orchestra.run.smithi076.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.47166.log.gz 2023-12-10T23:40:11.957 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40784.log 2023-12-10T23:40:11.958 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69123.log 2023-12-10T23:40:11.958 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.63496.log: /var/log/ceph/cluster2-client.mirror.40784.log: 4.7% -- replaced with /var/log/ceph/cluster2-client.mirror.40784.log.gz 2023-12-10T23:40:11.958 INFO:teuthology.orchestra.run.smithi076.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.63496.log.gz 2023-12-10T23:40:11.959 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71245.log 2023-12-10T23:40:11.960 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76385.log 2023-12-10T23:40:11.960 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.69123.log: /var/log/ceph/cluster1-client.mirror.71245.log: 86.7% 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.69123.log.gz 2023-12-10T23:40:11.960 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.71245.log.gz 2023-12-10T23:40:11.960 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63999.log 2023-12-10T23:40:11.960 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72825.log 2023-12-10T23:40:11.961 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.76385.log: /var/log/ceph/cluster1-client.mirror.63999.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.76385.log.gz 2023-12-10T23:40:11.962 INFO:teuthology.orchestra.run.smithi076.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.63999.log.gz 2023-12-10T23:40:11.962 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47876.log 2023-12-10T23:40:11.962 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.72825.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.72825.log.gz 2023-12-10T23:40:11.963 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71447.log 2023-12-10T23:40:11.963 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42326.log 2023-12-10T23:40:11.963 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.47876.log: /var/log/ceph/cluster1-client.mirror.71447.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.71447.log.gz 2023-12-10T23:40:11.963 INFO:teuthology.orchestra.run.smithi076.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47876.log.gz 2023-12-10T23:40:11.964 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44093.log 2023-12-10T23:40:11.965 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66430.log 2023-12-10T23:40:11.965 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.admin.42326.log: /var/log/ceph/cluster1-client.mirror.44093.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.42326.log.gz 2023-12-10T23:40:11.965 INFO:teuthology.orchestra.run.smithi076.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.44093.log.gz 2023-12-10T23:40:11.966 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53490.log 2023-12-10T23:40:11.966 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73644.log 2023-12-10T23:40:11.966 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.66430.log: /var/log/ceph/cluster1-client.mirror.53490.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53490.log.gz 2023-12-10T23:40:11.967 INFO:teuthology.orchestra.run.smithi076.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.66430.log.gz 2023-12-10T23:40:11.967 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57264.log 2023-12-10T23:40:11.967 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65545.log 2023-12-10T23:40:11.967 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.73644.log: /var/log/ceph/cluster1-client.mirror.57264.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.73644.log.gz 2023-12-10T23:40:11.968 INFO:teuthology.orchestra.run.smithi076.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.57264.log.gz 2023-12-10T23:40:11.968 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50020.log 2023-12-10T23:40:11.969 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68084.log 2023-12-10T23:40:11.969 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.65545.log: /var/log/ceph/cluster2-client.mirror.50020.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50020.log.gz 2023-12-10T23:40:11.969 INFO:teuthology.orchestra.run.smithi076.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.65545.log.gz 2023-12-10T23:40:11.969 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66740.log 2023-12-10T23:40:11.969 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59869.log 2023-12-10T23:40:11.970 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.68084.log: /var/log/ceph/cluster2-client.mirror.66740.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.66740.log.gz 2023-12-10T23:40:11.971 INFO:teuthology.orchestra.run.smithi076.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.68084.log.gz 2023-12-10T23:40:11.971 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44199.log 2023-12-10T23:40:11.971 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50718.log 2023-12-10T23:40:11.972 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.59869.log: /var/log/ceph/cluster2-client.mirror.44199.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59869.log.gz 2023-12-10T23:40:11.973 INFO:teuthology.orchestra.run.smithi076.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.44199.log.gz 2023-12-10T23:40:11.973 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46514.log 2023-12-10T23:40:11.973 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75091.log 2023-12-10T23:40:11.974 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.50718.log: /var/log/ceph/cluster1-client.mirror.46514.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50718.log.gz 2023-12-10T23:40:11.974 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71701.log 2023-12-10T23:40:11.974 INFO:teuthology.orchestra.run.smithi076.stderr: 91.3% -- replaced with /var/log/ceph/cluster1-client.mirror.46514.log.gz 2023-12-10T23:40:11.974 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.75091.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.75091.log.gz 2023-12-10T23:40:11.975 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67609.log 2023-12-10T23:40:11.975 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53985.log 2023-12-10T23:40:11.976 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.71701.log: /var/log/ceph/cluster2-client.mirror.67609.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.71701.log.gz 2023-12-10T23:40:11.976 INFO:teuthology.orchestra.run.smithi076.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.67609.log.gz 2023-12-10T23:40:11.976 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43089.log 2023-12-10T23:40:11.977 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.53985.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.53985.log.gz 2023-12-10T23:40:11.977 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59076.log 2023-12-10T23:40:11.977 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70030.log 2023-12-10T23:40:11.978 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.43089.log: /var/log/ceph/cluster2-client.mirror.59076.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.43089.log.gz 2023-12-10T23:40:11.978 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59188.log 2023-12-10T23:40:11.978 INFO:teuthology.orchestra.run.smithi076.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.59076.log.gz 2023-12-10T23:40:11.979 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.70030.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70030.log.gz 2023-12-10T23:40:11.979 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66513.log 2023-12-10T23:40:11.980 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43408.log 2023-12-10T23:40:11.980 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.59188.log: /var/log/ceph/cluster2-client.mirror.66513.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.59188.log.gz 2023-12-10T23:40:11.980 INFO:teuthology.orchestra.run.smithi076.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.66513.log.gz 2023-12-10T23:40:11.980 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56668.log 2023-12-10T23:40:11.981 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73497.log 2023-12-10T23:40:11.981 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.43408.log: /var/log/ceph/cluster2-client.mirror.56668.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.43408.log.gz 2023-12-10T23:40:11.982 INFO:teuthology.orchestra.run.smithi076.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56668.log.gz 2023-12-10T23:40:11.982 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70255.log 2023-12-10T23:40:11.982 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.73497.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.73497.log.gz 2023-12-10T23:40:11.983 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67298.log 2023-12-10T23:40:11.983 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78014.log 2023-12-10T23:40:11.984 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.70255.log: /var/log/ceph/cluster2-client.mirror.67298.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70255.log.gz 2023-12-10T23:40:11.984 INFO:teuthology.orchestra.run.smithi076.stderr: 87.8%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70570.log 2023-12-10T23:40:11.984 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.67298.log.gz 2023-12-10T23:40:11.985 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.78014.log: 67.9% -- replaced with /var/log/ceph/cluster1-client.mirror.78014.log.gz 2023-12-10T23:40:11.986 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49904.log 2023-12-10T23:40:11.986 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73101.log 2023-12-10T23:40:11.986 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.70570.log: /var/log/ceph/cluster2-client.mirror.49904.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.70570.log.gz 2023-12-10T23:40:11.986 INFO:teuthology.orchestra.run.smithi076.stderr: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49904.log.gz 2023-12-10T23:40:11.986 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64131.log 2023-12-10T23:40:11.987 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65793.log 2023-12-10T23:40:11.987 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.73101.log: /var/log/ceph/cluster2-client.mirror.64131.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.73101.log.gz 2023-12-10T23:40:11.988 INFO:teuthology.orchestra.run.smithi076.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.64131.log.gz 2023-12-10T23:40:11.988 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78200.log 2023-12-10T23:40:11.988 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60039.log 2023-12-10T23:40:11.989 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.65793.log: /var/log/ceph/cluster1-client.mirror.78200.log: 3.5% -- replaced with /var/log/ceph/cluster1-client.mirror.78200.log.gz 2023-12-10T23:40:11.989 INFO:teuthology.orchestra.run.smithi076.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.65793.log.gz 2023-12-10T23:40:11.989 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50171.log 2023-12-10T23:40:11.990 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70817.log 2023-12-10T23:40:11.990 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.50171.log: /var/log/ceph/cluster2-client.mirror.60039.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.60039.log.gz 2023-12-10T23:40:11.990 INFO:teuthology.orchestra.run.smithi076.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50171.log.gz 2023-12-10T23:40:11.991 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68147.log 2023-12-10T23:40:11.991 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60892.log 2023-12-10T23:40:11.991 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.70817.log: /var/log/ceph/cluster2-client.mirror.68147.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.70817.log.gz 2023-12-10T23:40:11.992 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54752.log 2023-12-10T23:40:11.993 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.60892.log: 90.0% 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60892.log.gz 2023-12-10T23:40:11.993 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.68147.log.gz 2023-12-10T23:40:11.993 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63651.log 2023-12-10T23:40:11.994 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62089.log 2023-12-10T23:40:11.994 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.54752.log: /var/log/ceph/cluster2-client.mirror.63651.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54752.log.gz 2023-12-10T23:40:11.994 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55141.log 2023-12-10T23:40:11.994 INFO:teuthology.orchestra.run.smithi076.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.63651.log.gz 2023-12-10T23:40:11.996 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.62089.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53036.log.gz 2023-12-10T23:40:11.996 INFO:teuthology.orchestra.run.smithi076.stderr: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.62089.log.gz 2023-12-10T23:40:11.996 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66161.log 2023-12-10T23:40:11.997 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.55141.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67919.log 2023-12-10T23:40:11.998 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.66161.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.55141.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61552.log 2023-12-10T23:40:11.998 INFO:teuthology.orchestra.run.smithi076.stderr: 2023-12-10T23:40:11.999 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.67919.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56761.log 2023-12-10T23:40:11.999 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.61552.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.67919.log.gz 2023-12-10T23:40:11.999 INFO:teuthology.orchestra.run.smithi076.stderr: 87.5% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.61552.log.gz 2023-12-10T23:40:11.999 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.66161.log.gz 2023-12-10T23:40:11.999 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67774.log 2023-12-10T23:40:12.000 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.56761.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52819.log 2023-12-10T23:40:12.001 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.67774.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56761.log.gz 2023-12-10T23:40:12.001 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47146.log 2023-12-10T23:40:12.001 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66306.log 2023-12-10T23:40:12.001 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.52819.log: /var/log/ceph/cluster2-client.mirror.47146.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.52819.log.gz 2023-12-10T23:40:12.001 INFO:teuthology.orchestra.run.smithi076.stderr: 90.1%gzip -- replaced with /var/log/ceph/cluster2-client.mirror.67774.log.gz 2023-12-10T23:40:12.001 INFO:teuthology.orchestra.run.smithi076.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67153.log 2023-12-10T23:40:12.002 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.66306.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.66306.log.gz 2023-12-10T23:40:12.002 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69782.log 2023-12-10T23:40:12.002 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46888.log 2023-12-10T23:40:12.003 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.67153.log: /var/log/ceph/cluster1-client.mirror.69782.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69782.log.gz 2023-12-10T23:40:12.003 INFO:teuthology.orchestra.run.smithi076.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.67153.log.gz 2023-12-10T23:40:12.004 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52441.log 2023-12-10T23:40:12.004 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59746.log 2023-12-10T23:40:12.005 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.46888.log: /var/log/ceph/cluster2-client.mirror.52441.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.52441.log.gz 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47146.log.gz 2023-12-10T23:40:12.005 INFO:teuthology.orchestra.run.smithi076.stderr: 2023-12-10T23:40:12.005 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64422.log 2023-12-10T23:40:12.006 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.59746.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66554.log 2023-12-10T23:40:12.006 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.64422.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.64422.log.gz 2023-12-10T23:40:12.007 INFO:teuthology.orchestra.run.smithi076.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59746.log.gz 2023-12-10T23:40:12.010 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.78217.log 2023-12-10T23:40:12.010 INFO:teuthology.orchestra.run.smithi076.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46888.log.gz 2023-12-10T23:40:12.010 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46759.log 2023-12-10T23:40:12.010 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.66554.log: /var/log/ceph/cluster1-client.admin.78217.log: 2.3% -- replaced with /var/log/ceph/cluster1-client.admin.78217.log.gz 2023-12-10T23:40:12.010 INFO:teuthology.orchestra.run.smithi076.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.66554.log.gz 2023-12-10T23:40:12.011 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78830.log 2023-12-10T23:40:12.011 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.46759.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63122.log 2023-12-10T23:40:12.012 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.78830.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40891.log 2023-12-10T23:40:12.012 INFO:teuthology.orchestra.run.smithi076.stderr: 9.2% -- replaced with /var/log/ceph/cluster2-client.mirror.78830.log.gz 2023-12-10T23:40:12.013 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66575.log 2023-12-10T23:40:12.013 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.63122.log: /var/log/ceph/cluster2-client.mirror.40891.log: 58.7% -- replaced with /var/log/ceph/cluster2-client.mirror.40891.log.gz 2023-12-10T23:40:12.013 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76669.log 2023-12-10T23:40:12.014 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.66575.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.66575.log.gz 2023-12-10T23:40:12.014 INFO:teuthology.orchestra.run.smithi076.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.63122.log.gz 2023-12-10T23:40:12.014 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65721.log 2023-12-10T23:40:12.015 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.76669.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64085.log 2023-12-10T23:40:12.015 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.65721.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.76669.log.gz 2023-12-10T23:40:12.016 INFO:teuthology.orchestra.run.smithi076.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46759.log.gz 2023-12-10T23:40:12.016 INFO:teuthology.orchestra.run.smithi076.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.65721.log.gz 2023-12-10T23:40:12.016 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53014.log 2023-12-10T23:40:12.018 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42010.log 2023-12-10T23:40:12.018 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.64085.log: /var/log/ceph/cluster2-client.mirror.53014.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71177.log 2023-12-10T23:40:12.018 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.42010.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53014.log.gz 2023-12-10T23:40:12.018 INFO:teuthology.orchestra.run.smithi076.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.42010.log.gz 2023-12-10T23:40:12.018 INFO:teuthology.orchestra.run.smithi076.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.64085.log.gz 2023-12-10T23:40:12.020 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63210.log 2023-12-10T23:40:12.020 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.71177.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46994.log 2023-12-10T23:40:12.020 INFO:teuthology.orchestra.run.smithi076.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71177.log.gz/var/log/ceph/cluster2-client.mirror.63210.log: 2023-12-10T23:40:12.020 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59309.log 2023-12-10T23:40:12.020 INFO:teuthology.orchestra.run.smithi076.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.63210.log.gz 2023-12-10T23:40:12.021 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64933.log 2023-12-10T23:40:12.021 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.46994.log: /var/log/ceph/cluster2-client.mirror.59309.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.59309.log.gz 2023-12-10T23:40:12.021 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46779.log 2023-12-10T23:40:12.022 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48451.log 2023-12-10T23:40:12.022 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.64933.log: /var/log/ceph/cluster1-client.mirror.46779.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.64933.log.gz 2023-12-10T23:40:12.022 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71132.log 2023-12-10T23:40:12.024 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.48451.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.46994.log.gz 2023-12-10T23:40:12.025 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60279.log 2023-12-10T23:40:12.026 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.71132.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71132.log.gz 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.46779.log.gz 2023-12-10T23:40:12.026 INFO:teuthology.orchestra.run.smithi076.stderr: 2023-12-10T23:40:12.027 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46014.log 2023-12-10T23:40:12.027 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77811.log 2023-12-10T23:40:12.028 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.60279.log: /var/log/ceph/cluster2-client.mirror.46014.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.60279.log.gz 2023-12-10T23:40:12.028 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65565.log 2023-12-10T23:40:12.029 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.77811.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.77811.log.gz 2023-12-10T23:40:12.029 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64666.log 2023-12-10T23:40:12.030 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.65565.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.65565.log.gz 2023-12-10T23:40:12.031 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66699.log 2023-12-10T23:40:12.031 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.64666.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.64666.log.gz 2023-12-10T23:40:12.032 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58852.log 2023-12-10T23:40:12.033 INFO:teuthology.orchestra.run.smithi076.stderr: 93.5%/var/log/ceph/cluster2-client.mirror.66699.log: -- replaced with /var/log/ceph/cluster2-client.mirror.46014.log.gz 2023-12-10T23:40:12.033 INFO:teuthology.orchestra.run.smithi076.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.66699.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58052.log 2023-12-10T23:40:12.033 INFO:teuthology.orchestra.run.smithi076.stderr: 2023-12-10T23:40:12.033 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.58852.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57783.log 2023-12-10T23:40:12.034 INFO:teuthology.orchestra.run.smithi076.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58852.log.gz 2023-12-10T23:40:12.035 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.58052.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58052.log.gz 2023-12-10T23:40:12.036 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39269.log 2023-12-10T23:40:12.037 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.78892.log 2023-12-10T23:40:12.037 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.57783.log: /var/log/ceph/cluster2-client.admin.39269.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39269.log.gz 2023-12-10T23:40:12.037 INFO:teuthology.orchestra.run.smithi076.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.57783.log.gz 2023-12-10T23:40:12.038 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74701.log 2023-12-10T23:40:12.038 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.admin.78892.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45393.log 2023-12-10T23:40:12.038 INFO:teuthology.orchestra.run.smithi076.stderr: 7.1% -- replaced with /var/log/ceph/cluster1-client.admin.78892.log.gz 2023-12-10T23:40:12.039 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.74701.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.74701.log.gz 2023-12-10T23:40:12.039 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67257.log 2023-12-10T23:40:12.039 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65432.log 2023-12-10T23:40:12.040 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.45393.log: /var/log/ceph/cluster2-client.mirror.67257.log: 89.2% -- replaced with /var/log/ceph/cluster1-client.mirror.45393.log.gz 2023-12-10T23:40:12.040 INFO:teuthology.orchestra.run.smithi076.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.67257.log.gz 2023-12-10T23:40:12.040 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71560.log 2023-12-10T23:40:12.040 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48785.log 2023-12-10T23:40:12.041 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.65432.log: /var/log/ceph/cluster1-client.mirror.71560.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.71560.log.gz 2023-12-10T23:40:12.041 INFO:teuthology.orchestra.run.smithi076.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.65432.log.gz 2023-12-10T23:40:12.041 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48728.log 2023-12-10T23:40:12.042 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66967.log 2023-12-10T23:40:12.042 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.48785.log: /var/log/ceph/cluster2-client.mirror.48728.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48785.log.gz 2023-12-10T23:40:12.042 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47616.log 2023-12-10T23:40:12.044 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.66967.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.66967.log.gz 2023-12-10T23:40:12.044 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58094.log 2023-12-10T23:40:12.045 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.47616.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.47616.log.gz 2023-12-10T23:40:12.046 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47232.log 2023-12-10T23:40:12.047 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.58094.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58094.log.gz 2023-12-10T23:40:12.047 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57287.log 2023-12-10T23:40:12.051 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.47232.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.44630.log.gz 2023-12-10T23:40:12.052 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52720.log 2023-12-10T23:40:12.053 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.57287.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.57287.log.gz 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47232.log.gz 2023-12-10T23:40:12.053 INFO:teuthology.orchestra.run.smithi076.stderr: 2023-12-10T23:40:12.053 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46694.log 2023-12-10T23:40:12.054 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63605.log 2023-12-10T23:40:12.054 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.52720.log: /var/log/ceph/cluster1-client.mirror.46694.log: 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.46694.log.gz 2023-12-10T23:40:12.055 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66926.log 2023-12-10T23:40:12.056 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.63605.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.63605.log.gz 2023-12-10T23:40:12.056 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39626.log 2023-12-10T23:40:12.057 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.66926.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.66926.log.gz 2023-12-10T23:40:12.057 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52215.log 2023-12-10T23:40:12.058 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.admin.39626.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39626.log.gz 2023-12-10T23:40:12.061 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75448.log 2023-12-10T23:40:12.078 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.52215.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63233.log 2023-12-10T23:40:12.078 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.75448.log: 87.1% -- replaced with /var/log/ceph/cluster2-client.mirror.75448.log.gz 2023-12-10T23:40:12.085 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58480.log 2023-12-10T23:40:12.086 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.63233.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.63233.log.gz 2023-12-10T23:40:12.105 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49274.log 2023-12-10T23:40:12.107 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.58480.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58480.log.gz 2023-12-10T23:40:12.121 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71042.log 2023-12-10T23:40:12.123 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.49274.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49274.log.gz 2023-12-10T23:40:12.123 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69335.log 2023-12-10T23:40:12.124 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.71042.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.71042.log.gz 2023-12-10T23:40:12.124 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50893.log 2023-12-10T23:40:12.125 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.69335.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.69335.log.gz 2023-12-10T23:40:12.125 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43978.log 2023-12-10T23:40:12.126 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.50893.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50893.log.gz 2023-12-10T23:40:12.127 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48042.log 2023-12-10T23:40:12.127 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.43978.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.43978.log.gz 2023-12-10T23:40:12.128 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67485.log 2023-12-10T23:40:12.128 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.48042.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.48042.log.gz 2023-12-10T23:40:12.129 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43500.log 2023-12-10T23:40:12.130 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.67485.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.67485.log.gz 2023-12-10T23:40:12.130 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70997.log 2023-12-10T23:40:12.131 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.43500.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.43500.log.gz 2023-12-10T23:40:12.131 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65949.log 2023-12-10T23:40:12.132 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.70997.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70997.log.gz 2023-12-10T23:40:12.133 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43636.log 2023-12-10T23:40:12.135 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.65949.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65949.log.gz 2023-12-10T23:40:12.135 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60823.log 2023-12-10T23:40:12.141 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.43636.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43636.log.gz 2023-12-10T23:40:12.141 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40622.log 2023-12-10T23:40:12.142 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.60823.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60823.log.gz 2023-12-10T23:40:12.143 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58782.log 2023-12-10T23:40:12.143 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.admin.40622.log: 67.9% -- replaced with /var/log/ceph/cluster2-client.admin.40622.log.gz 2023-12-10T23:40:12.144 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62878.log 2023-12-10T23:40:12.144 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.58782.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58782.log.gz 2023-12-10T23:40:12.145 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57466.log 2023-12-10T23:40:12.146 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.62878.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52045.log.gz 2023-12-10T23:40:12.147 INFO:teuthology.orchestra.run.smithi076.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62878.log.gz 2023-12-10T23:40:12.147 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76488.log 2023-12-10T23:40:12.148 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60621.log 2023-12-10T23:40:12.148 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.57466.log: /var/log/ceph/cluster2-client.mirror.76488.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.57466.log.gz 2023-12-10T23:40:12.149 INFO:teuthology.orchestra.run.smithi076.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.76488.log.gz 2023-12-10T23:40:12.149 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75407.log 2023-12-10T23:40:12.149 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.60621.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60621.log.gz 2023-12-10T23:40:12.150 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57824.log 2023-12-10T23:40:12.150 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.75407.log: 87.1% -- replaced with /var/log/ceph/cluster2-client.mirror.75407.log.gz 2023-12-10T23:40:12.151 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69102.log 2023-12-10T23:40:12.153 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.57824.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.57824.log.gz 2023-12-10T23:40:12.153 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64844.log 2023-12-10T23:40:12.154 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.69102.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.69102.log.gz 2023-12-10T23:40:12.158 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.78327.log 2023-12-10T23:40:12.158 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.64844.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.64844.log.gz 2023-12-10T23:40:12.158 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45878.log 2023-12-10T23:40:12.158 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.admin.78327.log: 2.3% -- replaced with /var/log/ceph/cluster1-client.admin.78327.log.gz 2023-12-10T23:40:12.159 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55458.log 2023-12-10T23:40:12.159 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.45878.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.45878.log.gz 2023-12-10T23:40:12.160 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39776.log 2023-12-10T23:40:12.160 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.55458.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.55458.log.gz 2023-12-10T23:40:12.160 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51772.log 2023-12-10T23:40:12.161 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.admin.39776.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39776.log.gz 2023-12-10T23:40:12.161 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59209.log 2023-12-10T23:40:12.162 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.51772.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51772.log.gz 2023-12-10T23:40:12.162 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74910.log 2023-12-10T23:40:12.162 INFO:teuthology.orchestra.run.smithi076.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50610.log.gz 2023-12-10T23:40:12.164 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.59209.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.59209.log.gz 2023-12-10T23:40:12.165 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75346.log 2023-12-10T23:40:12.165 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50676.log 2023-12-10T23:40:12.166 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.74910.log: /var/log/ceph/cluster2-client.mirror.75346.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.75346.log.gz 87.1% -- replaced with /var/log/ceph/cluster2-client.mirror.74910.log.gz 2023-12-10T23:40:12.167 INFO:teuthology.orchestra.run.smithi076.stderr: 2023-12-10T23:40:12.167 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57598.log 2023-12-10T23:40:12.167 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61159.log 2023-12-10T23:40:12.168 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.50676.log: /var/log/ceph/cluster2-client.mirror.57598.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50676.log.gz 2023-12-10T23:40:12.168 INFO:teuthology.orchestra.run.smithi076.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57598.log.gz 2023-12-10T23:40:12.168 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47252.log 2023-12-10T23:40:12.169 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.61159.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.61159.log.gz 2023-12-10T23:40:12.169 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62966.log 2023-12-10T23:40:12.169 INFO:teuthology.orchestra.run.smithi076.stderr: 93.8%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70975.log 2023-12-10T23:40:12.169 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.45567.log.gz 2023-12-10T23:40:12.170 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.47252.log: /var/log/ceph/cluster2-client.mirror.62966.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.62966.log.gz 2023-12-10T23:40:12.174 INFO:teuthology.orchestra.run.smithi076.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.47252.log.gz 2023-12-10T23:40:12.175 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55675.log 2023-12-10T23:40:12.175 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.70975.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54333.log 2023-12-10T23:40:12.175 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.55675.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58434.log 2023-12-10T23:40:12.175 INFO:teuthology.orchestra.run.smithi076.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70975.log.gz 2023-12-10T23:40:12.176 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.54333.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.55675.log.gz 2023-12-10T23:40:12.176 INFO:teuthology.orchestra.run.smithi076.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.54333.log.gz 2023-12-10T23:40:12.176 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62348.log 2023-12-10T23:40:12.177 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.58434.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74931.log 2023-12-10T23:40:12.178 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.62348.log: 87.7%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63671.log 2023-12-10T23:40:12.178 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.58434.log.gz 2023-12-10T23:40:12.178 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.74931.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.62348.log.gz 2023-12-10T23:40:12.178 INFO:teuthology.orchestra.run.smithi076.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.74931.log.gz 2023-12-10T23:40:12.179 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62394.log 2023-12-10T23:40:12.179 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.63671.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61377.log 2023-12-10T23:40:12.180 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.62394.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41506.log 2023-12-10T23:40:12.180 INFO:teuthology.orchestra.run.smithi076.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.63671.log.gz 2023-12-10T23:40:12.180 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.61377.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.62394.log.gz 2023-12-10T23:40:12.180 INFO:teuthology.orchestra.run.smithi076.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.61377.log.gz 2023-12-10T23:40:12.180 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46908.log 2023-12-10T23:40:12.181 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.admin.41506.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47716.log 2023-12-10T23:40:12.181 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.46908.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76631.log 2023-12-10T23:40:12.181 INFO:teuthology.orchestra.run.smithi076.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.41506.log.gz 2023-12-10T23:40:12.181 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.47716.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.47716.log.gz 2023-12-10T23:40:12.182 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68875.log 2023-12-10T23:40:12.182 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.76631.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51342.log 2023-12-10T23:40:12.183 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.68875.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.68875.log.gz 2023-12-10T23:40:12.184 INFO:teuthology.orchestra.run.smithi076.stderr: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.76631.log.gz 2023-12-10T23:40:12.184 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51379.log 2023-12-10T23:40:12.184 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67464.log 2023-12-10T23:40:12.184 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.51342.log: /var/log/ceph/cluster2-client.mirror.51379.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51379.log.gz 2023-12-10T23:40:12.185 INFO:teuthology.orchestra.run.smithi076.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.46908.log.gz 2023-12-10T23:40:12.185 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77190.log 2023-12-10T23:40:12.186 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.67464.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78670.log 2023-12-10T23:40:12.186 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.77190.log: 62.7% -- replaced with /var/log/ceph/cluster2-client.mirror.77190.log.gz 2023-12-10T23:40:12.187 INFO:teuthology.orchestra.run.smithi076.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.67464.log.gz 2023-12-10T23:40:12.187 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57737.log 2023-12-10T23:40:12.188 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.78670.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70660.log 2023-12-10T23:40:12.188 INFO:teuthology.orchestra.run.smithi076.stderr: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.78670.log.gz 2023-12-10T23:40:12.188 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.57737.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42950.log 2023-12-10T23:40:12.189 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.70660.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.57737.log.gz 2023-12-10T23:40:12.189 INFO:teuthology.orchestra.run.smithi076.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70660.log.gz 2023-12-10T23:40:12.190 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39348.log 2023-12-10T23:40:12.190 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.42950.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.42950.log.gz 2023-12-10T23:40:12.190 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65611.log 2023-12-10T23:40:12.191 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59289.log 2023-12-10T23:40:12.191 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.admin.39348.log: /var/log/ceph/cluster1-client.mirror.65611.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39348.log.gz 2023-12-10T23:40:12.191 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77870.log 2023-12-10T23:40:12.191 INFO:teuthology.orchestra.run.smithi076.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.65611.log.gz 2023-12-10T23:40:12.192 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.59289.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.59289.log.gz 2023-12-10T23:40:12.192 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47361.log 2023-12-10T23:40:12.193 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66988.log 2023-12-10T23:40:12.193 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.77870.log: /var/log/ceph/cluster2-client.mirror.47361.log: 49.4% -- replaced with /var/log/ceph/cluster1-client.mirror.77870.log.gz 2023-12-10T23:40:12.194 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44480.log 2023-12-10T23:40:12.195 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.66988.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.66988.log.gz 2023-12-10T23:40:12.195 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.46514.log 2023-12-10T23:40:12.196 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.44480.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.44480.log.gz 2023-12-10T23:40:12.196 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39676.log 2023-12-10T23:40:12.198 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.2.46514.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47361.log.gz 2023-12-10T23:40:12.206 INFO:teuthology.orchestra.run.smithi076.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.2.46514.log.gz 2023-12-10T23:40:12.206 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57195.log 2023-12-10T23:40:12.207 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75195.log 2023-12-10T23:40:12.208 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.admin.39676.log: /var/log/ceph/cluster2-client.mirror.57195.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39676.log.gz 2023-12-10T23:40:12.208 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44299.log 2023-12-10T23:40:12.208 INFO:teuthology.orchestra.run.smithi076.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57195.log.gz 2023-12-10T23:40:12.209 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.75195.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.75195.log.gz 2023-12-10T23:40:12.209 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51204.log 2023-12-10T23:40:12.209 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66265.log 2023-12-10T23:40:12.210 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.44299.log: /var/log/ceph/cluster2-client.mirror.51204.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51204.log.gz 2023-12-10T23:40:12.210 INFO:teuthology.orchestra.run.smithi076.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.44299.log.gz 2023-12-10T23:40:12.210 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53325.log 2023-12-10T23:40:12.211 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55606.log 2023-12-10T23:40:12.212 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.66265.log: /var/log/ceph/cluster2-client.mirror.53325.log: 79.9% 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.66265.log.gz 2023-12-10T23:40:12.212 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.53325.log.gz 2023-12-10T23:40:12.213 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66451.log 2023-12-10T23:40:12.213 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75033.log 2023-12-10T23:40:12.213 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.55606.log: /var/log/ceph/cluster2-client.mirror.66451.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.55606.log.gz 2023-12-10T23:40:12.213 INFO:teuthology.orchestra.run.smithi076.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.66451.log.gz 2023-12-10T23:40:12.213 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55075.log 2023-12-10T23:40:12.214 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.75033.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.75033.log.gz 2023-12-10T23:40:12.214 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62654.log 2023-12-10T23:40:12.215 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66657.log 2023-12-10T23:40:12.215 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.55075.log: /var/log/ceph/cluster1-client.mirror.62654.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62654.log.gz 2023-12-10T23:40:12.215 INFO:teuthology.orchestra.run.smithi076.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.55075.log.gz 2023-12-10T23:40:12.216 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70682.log 2023-12-10T23:40:12.216 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47856.log 2023-12-10T23:40:12.217 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.66657.log: /var/log/ceph/cluster1-client.mirror.70682.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70682.log.gz 2023-12-10T23:40:12.217 INFO:teuthology.orchestra.run.smithi076.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.66657.log.gz 2023-12-10T23:40:12.218 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71267.log 2023-12-10T23:40:12.219 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65169.log 2023-12-10T23:40:12.219 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.47856.log: /var/log/ceph/cluster1-client.mirror.71267.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.71267.log.gz 2023-12-10T23:40:12.219 INFO:teuthology.orchestra.run.smithi076.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.47856.log.gz 2023-12-10T23:40:12.219 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42100.log 2023-12-10T23:40:12.220 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76841.log 2023-12-10T23:40:12.220 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.65169.log: /var/log/ceph/cluster2-client.mirror.42100.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.65169.log.gz 2023-12-10T23:40:12.220 INFO:teuthology.orchestra.run.smithi076.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.42100.log.gz 2023-12-10T23:40:12.221 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70727.log 2023-12-10T23:40:12.222 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57051.log 2023-12-10T23:40:12.222 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.76841.log: /var/log/ceph/cluster1-client.mirror.70727.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70727.log.gz 2023-12-10T23:40:12.222 INFO:teuthology.orchestra.run.smithi076.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.76841.log.gz 2023-12-10T23:40:12.222 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72625.log 2023-12-10T23:40:12.223 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57535.log 2023-12-10T23:40:12.223 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.57051.log: /var/log/ceph/cluster1-client.mirror.72625.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.57051.log.gz 2023-12-10T23:40:12.223 INFO:teuthology.orchestra.run.smithi076.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.72625.log.gz 2023-12-10T23:40:12.223 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75701.log 2023-12-10T23:40:12.223 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77942.log 2023-12-10T23:40:12.224 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.57535.log: /var/log/ceph/cluster2-client.mirror.75701.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.57535.log.gz 2023-12-10T23:40:12.225 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67401.log 2023-12-10T23:40:12.225 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.77942.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.77942.log.gz 2023-12-10T23:40:12.225 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45993.log 2023-12-10T23:40:12.226 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.67401.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.67401.log.gz 2023-12-10T23:40:12.227 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68833.log 2023-12-10T23:40:12.228 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.45993.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.45993.log.gz 2023-12-10T23:40:12.228 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56094.log 2023-12-10T23:40:12.230 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.68833.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.68833.log.gz 2023-12-10T23:40:12.230 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68895.log 2023-12-10T23:40:12.231 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.56094.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.56094.log.gz 2023-12-10T23:40:12.231 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52957.log 2023-12-10T23:40:12.232 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.68895.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.68895.log.gz 2023-12-10T23:40:12.233 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68064.log 2023-12-10T23:40:12.235 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.52957.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52720.log.gz 2023-12-10T23:40:12.237 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51522.log 2023-12-10T23:40:12.238 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.68064.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.68064.log.gz 2023-12-10T23:40:12.239 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43699.log 2023-12-10T23:40:12.254 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.51522.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44586.log 2023-12-10T23:40:12.259 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.43699.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43699.log.gz 2023-12-10T23:40:12.265 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44275.log 2023-12-10T23:40:12.266 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.44586.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.44586.log.gz 2023-12-10T23:40:12.268 INFO:teuthology.orchestra.run.smithi076.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51522.log.gz 2023-12-10T23:40:12.268 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54031.log 2023-12-10T23:40:12.269 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.44275.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.44275.log.gz 2023-12-10T23:40:12.269 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71380.log 2023-12-10T23:40:12.270 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.54031.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.54031.log.gz 2023-12-10T23:40:12.271 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40869.log 2023-12-10T23:40:12.272 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.71380.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71380.log.gz 2023-12-10T23:40:12.272 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69467.log 2023-12-10T23:40:12.273 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.40869.log: 60.8% -- replaced with /var/log/ceph/cluster2-client.mirror.40869.log.gz 2023-12-10T23:40:12.273 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59726.log 2023-12-10T23:40:12.274 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.69467.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.69467.log.gz 2023-12-10T23:40:12.274 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68280.log 2023-12-10T23:40:12.276 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.59726.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59726.log.gz 2023-12-10T23:40:12.277 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55479.log 2023-12-10T23:40:12.277 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.68280.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.68280.log.gz 2023-12-10T23:40:12.278 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63012.log 2023-12-10T23:40:12.280 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.55479.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.55479.log.gz 2023-12-10T23:40:12.280 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68321.log 2023-12-10T23:40:12.281 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.63012.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.63012.log.gz 2023-12-10T23:40:12.282 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48106.log 2023-12-10T23:40:12.282 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.68321.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.68321.log.gz 2023-12-10T23:40:12.283 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60644.log 2023-12-10T23:40:12.286 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.48106.log: 94.9% -- replaced with /var/log/ceph/cluster1-client.mirror.48106.log.gz 2023-12-10T23:40:12.286 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44501.log 2023-12-10T23:40:12.287 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.60644.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60644.log.gz 2023-12-10T23:40:12.287 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78567.log 2023-12-10T23:40:12.293 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.44501.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44501.log.gz 2023-12-10T23:40:12.293 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54143.log 2023-12-10T23:40:12.294 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.78567.log: 7.1% -- replaced with /var/log/ceph/cluster2-client.mirror.78567.log.gz 2023-12-10T23:40:12.294 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39347.log 2023-12-10T23:40:12.295 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.54143.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.54143.log.gz 2023-12-10T23:40:12.295 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47576.log 2023-12-10T23:40:12.296 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.admin.39347.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39347.log.gz 2023-12-10T23:40:12.296 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54310.log 2023-12-10T23:40:12.297 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.47576.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.47576.log.gz 2023-12-10T23:40:12.298 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63786.log 2023-12-10T23:40:12.299 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.54310.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.54310.log.gz 2023-12-10T23:40:12.299 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40190.log 2023-12-10T23:40:12.300 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.63786.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.63786.log.gz 2023-12-10T23:40:12.300 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62858.log 2023-12-10T23:40:12.301 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.admin.40190.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.40190.log.gz 2023-12-10T23:40:12.301 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76057.log 2023-12-10T23:40:12.304 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.62858.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.62858.log.gz 2023-12-10T23:40:12.304 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53283.log 2023-12-10T23:40:12.305 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.76057.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.76057.log.gz 2023-12-10T23:40:12.305 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50462.log 2023-12-10T23:40:12.306 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.53283.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53283.log.gz 2023-12-10T23:40:12.306 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50154.log 2023-12-10T23:40:12.314 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.50462.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57373.log 2023-12-10T23:40:12.316 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.50154.log: 79.8% 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.50154.log.gz 2023-12-10T23:40:12.316 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.50462.log.gz 2023-12-10T23:40:12.316 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78361.log 2023-12-10T23:40:12.317 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.57373.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57373.log.gz 2023-12-10T23:40:12.317 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54420.log 2023-12-10T23:40:12.318 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.78361.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78361.log.gz 2023-12-10T23:40:12.318 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76426.log 2023-12-10T23:40:12.321 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.54420.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.54420.log.gz 2023-12-10T23:40:12.321 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51478.log 2023-12-10T23:40:12.322 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.76426.log: 87.1% -- replaced with /var/log/ceph/cluster2-client.mirror.76426.log.gz 2023-12-10T23:40:12.322 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50755.log 2023-12-10T23:40:12.323 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.51478.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.51478.log.gz 2023-12-10T23:40:12.323 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62723.log 2023-12-10T23:40:12.324 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.50755.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50755.log.gz 2023-12-10T23:40:12.325 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58265.log 2023-12-10T23:40:12.325 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.62723.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.62723.log.gz 2023-12-10T23:40:12.326 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67733.log 2023-12-10T23:40:12.328 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.58265.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.58265.log.gz 2023-12-10T23:40:12.328 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70210.log 2023-12-10T23:40:12.329 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.67733.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.67733.log.gz 2023-12-10T23:40:12.330 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66285.log 2023-12-10T23:40:12.330 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.70210.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.70210.log.gz 2023-12-10T23:40:12.331 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40926.log 2023-12-10T23:40:12.333 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.66285.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.66285.log.gz 2023-12-10T23:40:12.333 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59251.log 2023-12-10T23:40:12.334 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.40926.log: 53.8% -- replaced with /var/log/ceph/cluster2-client.mirror.40926.log.gz 2023-12-10T23:40:12.334 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50935.log 2023-12-10T23:40:12.335 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.59251.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59251.log.gz 2023-12-10T23:40:12.336 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40041.log 2023-12-10T23:40:12.349 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.50935.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78577.log 2023-12-10T23:40:12.350 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.40041.log: 82.0% -- replaced with /var/log/ceph/cluster1-client.mirror.40041.log.gz 2023-12-10T23:40:12.365 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51263.log 2023-12-10T23:40:12.366 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.78577.log: 9.2% -- replaced with /var/log/ceph/cluster2-client.mirror.78577.log.gz 2023-12-10T23:40:12.367 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72969.log 2023-12-10T23:40:12.373 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.51263.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51342.log.gz 2023-12-10T23:40:12.373 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49558.log 2023-12-10T23:40:12.374 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.72969.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53513.log 2023-12-10T23:40:12.374 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.49558.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.72969.log.gz 2023-12-10T23:40:12.390 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49360.log 2023-12-10T23:40:12.391 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.53513.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53513.log.gz 2023-12-10T23:40:12.401 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70525.log 2023-12-10T23:40:12.402 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.49360.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49360.log.gz 2023-12-10T23:40:12.417 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46802.log 2023-12-10T23:40:12.418 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.70525.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70525.log.gz 2023-12-10T23:40:12.425 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66678.log 2023-12-10T23:40:12.433 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.46802.log: 93.5% 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52215.log.gz 2023-12-10T23:40:12.433 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.48106.log 2023-12-10T23:40:12.434 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.66678.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.66678.log.gz 2023-12-10T23:40:12.435 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47956.log 2023-12-10T23:40:12.445 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.1.48106.log: -- replaced with /var/log/ceph/cluster2-client.mirror.46802.log.gz 2023-12-10T23:40:12.446 INFO:teuthology.orchestra.run.smithi076.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48451.log.gz 2023-12-10T23:40:12.446 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49797.log 2023-12-10T23:40:12.447 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.47956.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47956.log.gz 2023-12-10T23:40:12.447 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47381.log 2023-12-10T23:40:12.457 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.49797.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63035.log 2023-12-10T23:40:12.470 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.47381.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.47381.log.gz 2023-12-10T23:40:12.470 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77302.log 2023-12-10T23:40:12.470 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.63035.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.63035.log.gz 2023-12-10T23:40:12.473 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65363.log 2023-12-10T23:40:12.474 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.77302.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.77302.log.gz 2023-12-10T23:40:12.486 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48649.log 2023-12-10T23:40:12.486 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.65363.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.65363.log.gz 2023-12-10T23:40:12.489 INFO:teuthology.orchestra.run.smithi076.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48728.log.gz 2023-12-10T23:40:12.489 INFO:teuthology.orchestra.run.smithi076.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.1.48106.log.gz 2023-12-10T23:40:12.490 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49294.log 2023-12-10T23:40:12.492 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.48649.log: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.49558.log.gz 2023-12-10T23:40:12.497 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48686.log 2023-12-10T23:40:12.498 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.49294.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49294.log.gz 2023-12-10T23:40:12.499 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40214.log 2023-12-10T23:40:12.500 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.48686.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48686.log.gz 2023-12-10T23:40:12.500 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48539.log 2023-12-10T23:40:12.501 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.admin.40214.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.40214.log.gz 2023-12-10T23:40:12.501 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47490.log 2023-12-10T23:40:12.517 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.48539.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67940.log 2023-12-10T23:40:12.523 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.47490.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47490.log.gz 2023-12-10T23:40:12.525 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53663.log 2023-12-10T23:40:12.526 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.67940.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.67940.log.gz 2023-12-10T23:40:12.542 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52351.log 2023-12-10T23:40:12.542 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.53663.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53663.log.gz 2023-12-10T23:40:12.549 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.62748.log 2023-12-10T23:40:12.550 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.52351.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52351.log.gz 2023-12-10T23:40:12.565 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76467.log 2023-12-10T23:40:12.573 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.1.62748.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45460.log 2023-12-10T23:40:12.574 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.76467.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.76467.log.gz 2023-12-10T23:40:12.581 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46362.log 2023-12-10T23:40:12.589 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.45460.log: 93.5%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58221.log 2023-12-10T23:40:12.590 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.46362.log: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.46362.log.gz 2023-12-10T23:40:12.598 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.45460.log.gz 2023-12-10T23:40:12.601 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69827.log 2023-12-10T23:40:12.602 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.58221.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58221.log.gz 2023-12-10T23:40:12.617 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46736.log 2023-12-10T23:40:12.618 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.69827.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.69827.log.gz 2023-12-10T23:40:12.628 INFO:teuthology.orchestra.run.smithi076.stderr: 93.8%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53686.log 2023-12-10T23:40:12.634 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.46736.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.52957.log.gz 2023-12-10T23:40:12.634 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.46736.log.gz 2023-12-10T23:40:12.634 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77026.log 2023-12-10T23:40:12.635 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.53686.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.53686.log.gz 2023-12-10T23:40:12.635 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66864.log 2023-12-10T23:40:12.636 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.77026.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.77026.log.gz 2023-12-10T23:40:12.636 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48706.log 2023-12-10T23:40:12.637 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.66864.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.66864.log.gz 2023-12-10T23:40:12.637 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66616.log 2023-12-10T23:40:12.638 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.48706.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48706.log.gz 2023-12-10T23:40:12.638 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69377.log 2023-12-10T23:40:12.639 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.66616.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.66616.log.gz 2023-12-10T23:40:12.640 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75601.log 2023-12-10T23:40:12.640 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.69377.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.69377.log.gz 2023-12-10T23:40:12.641 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67795.log 2023-12-10T23:40:12.641 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.75601.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.75601.log.gz 2023-12-10T23:40:12.642 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69356.log 2023-12-10T23:40:12.642 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.67795.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.67795.log.gz 2023-12-10T23:40:12.643 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58388.log 2023-12-10T23:40:12.643 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.69356.log: 2.6% -- replaced with /var/log/ceph/cluster1-client.mirror.69356.log.gz 2023-12-10T23:40:12.644 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59502.log 2023-12-10T23:40:12.644 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.58388.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.58388.log.gz 2023-12-10T23:40:12.645 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56203.log 2023-12-10T23:40:12.646 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.59502.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.59502.log.gz 2023-12-10T23:40:12.646 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51871.log 2023-12-10T23:40:12.647 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.56203.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56203.log.gz 2023-12-10T23:40:12.647 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39701.log 2023-12-10T23:40:12.649 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.51871.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.51871.log.gz 2023-12-10T23:40:12.649 INFO:teuthology.orchestra.run.smithi076.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49797.log.gz 2023-12-10T23:40:12.649 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56582.log 2023-12-10T23:40:12.649 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.admin.39701.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39701.log.gz 2023-12-10T23:40:12.650 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.49686.log 2023-12-10T23:40:12.650 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.56582.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.56582.log.gz 2023-12-10T23:40:12.654 INFO:teuthology.orchestra.run.smithi076.stderr: 92.8% -- replaced with /var/log/ceph/cluster2-client.mirror.1.62748.log.gz 2023-12-10T23:40:12.654 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59826.log 2023-12-10T23:40:12.655 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.2.49686.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68169.log 2023-12-10T23:40:12.656 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.59826.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.59826.log.gz 2023-12-10T23:40:12.668 INFO:teuthology.orchestra.run.smithi076.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.2.49686.log.gz 2023-12-10T23:40:12.668 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70885.log 2023-12-10T23:40:12.669 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.68169.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63542.log 2023-12-10T23:40:12.669 INFO:teuthology.orchestra.run.smithi076.stderr: 89.5%/var/log/ceph/cluster1-client.mirror.70885.log: -- replaced with /var/log/ceph/cluster1-client.mirror.68169.log.gz 2023-12-10T23:40:12.670 INFO:teuthology.orchestra.run.smithi076.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.70885.log.gz 2023-12-10T23:40:12.670 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67630.log 2023-12-10T23:40:12.671 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.63542.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40374.log 2023-12-10T23:40:12.671 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.67630.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.63542.log.gz 2023-12-10T23:40:12.671 INFO:teuthology.orchestra.run.smithi076.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.67630.log.gz 2023-12-10T23:40:12.671 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55389.log 2023-12-10T23:40:12.671 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43592.log 2023-12-10T23:40:12.672 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.40374.log: /var/log/ceph/cluster1-client.mirror.55389.log: 54.2% -- replaced with /var/log/ceph/cluster1-client.mirror.40374.log.gz 2023-12-10T23:40:12.673 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77743.log 2023-12-10T23:40:12.673 INFO:teuthology.orchestra.run.smithi076.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.55389.log.gz 2023-12-10T23:40:12.673 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.43592.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.43592.log.gz 2023-12-10T23:40:12.674 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72691.log 2023-12-10T23:40:12.674 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68730.log 2023-12-10T23:40:12.674 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.77743.log: /var/log/ceph/cluster2-client.mirror.72691.log: 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.77743.log.gz 2023-12-10T23:40:12.674 INFO:teuthology.orchestra.run.smithi076.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.72691.log.gz 2023-12-10T23:40:12.674 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44434.log 2023-12-10T23:40:12.676 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77464.log 2023-12-10T23:40:12.676 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.68730.log: /var/log/ceph/cluster1-client.mirror.44434.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.68730.log.gz 2023-12-10T23:40:12.676 INFO:teuthology.orchestra.run.smithi076.stderr: 88.8% -- replaced with /var/log/ceph/cluster1-client.mirror.44434.log.gz 2023-12-10T23:40:12.676 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77087.log 2023-12-10T23:40:12.676 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.40041.log 2023-12-10T23:40:12.677 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.77464.log: /var/log/ceph/cluster2-client.mirror.77087.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.77464.log.gz 2023-12-10T23:40:12.677 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60731.log 2023-12-10T23:40:12.677 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.3.40041.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.77087.log.gz 2023-12-10T23:40:12.679 INFO:teuthology.orchestra.run.smithi076.stderr: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.3.40041.log.gz 2023-12-10T23:40:12.679 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78658.log 2023-12-10T23:40:12.680 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43523.log 2023-12-10T23:40:12.680 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.60731.log: /var/log/ceph/cluster2-client.mirror.78658.log: 10.2% -- replaced with /var/log/ceph/cluster2-client.mirror.78658.log.gz 2023-12-10T23:40:12.682 INFO:teuthology.orchestra.run.smithi076.stderr: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.60731.log.gz 2023-12-10T23:40:12.682 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70795.log 2023-12-10T23:40:12.683 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70367.log 2023-12-10T23:40:12.683 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.43523.log: /var/log/ceph/cluster1-client.mirror.70795.log: 86.8% 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.43523.log.gz 2023-12-10T23:40:12.683 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.70795.log.gz 2023-12-10T23:40:12.684 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49138.log 2023-12-10T23:40:12.684 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55118.log 2023-12-10T23:40:12.685 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.70367.log: /var/log/ceph/cluster2-client.mirror.49138.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70367.log.gz 2023-12-10T23:40:12.685 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51300.log 2023-12-10T23:40:12.685 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.55118.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.55118.log.gz 2023-12-10T23:40:12.686 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63407.log 2023-12-10T23:40:12.687 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.51300.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51300.log.gz 2023-12-10T23:40:12.687 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50381.log 2023-12-10T23:40:12.688 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.63407.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63407.log.gz 2023-12-10T23:40:12.689 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51678.log 2023-12-10T23:40:12.689 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.50381.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50381.log.gz 2023-12-10T23:40:12.690 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63343.log 2023-12-10T23:40:12.704 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.51678.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.75701.log.gz 2023-12-10T23:40:12.705 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60299.log 2023-12-10T23:40:12.708 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.63343.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.63343.log.gz 2023-12-10T23:40:12.708 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45970.log 2023-12-10T23:40:12.709 INFO:teuthology.orchestra.run.smithi076.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50935.log.gz 2023-12-10T23:40:12.709 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.60299.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60299.log.gz 2023-12-10T23:40:12.709 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52878.log 2023-12-10T23:40:12.710 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64756.log 2023-12-10T23:40:12.710 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.45970.log: /var/log/ceph/cluster2-client.mirror.52878.log: 89.2% -- replaced with /var/log/ceph/cluster1-client.mirror.45970.log.gz 2023-12-10T23:40:12.710 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73664.log 2023-12-10T23:40:12.713 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.64756.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.64756.log.gz 2023-12-10T23:40:12.713 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75787.log 2023-12-10T23:40:12.714 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.73664.log: 87.1% -- replaced with /var/log/ceph/cluster2-client.mirror.73664.log.gz 2023-12-10T23:40:12.714 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62989.log 2023-12-10T23:40:12.715 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.75787.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.75787.log.gz 2023-12-10T23:40:12.716 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78903.log 2023-12-10T23:40:12.716 INFO:teuthology.orchestra.run.smithi076.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51263.log.gz 2023-12-10T23:40:12.717 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.62989.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74303.log 2023-12-10T23:40:12.717 INFO:teuthology.orchestra.run.smithi076.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62989.log.gz 2023-12-10T23:40:12.717 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.admin.78903.log: 9.2% -- replaced with /var/log/ceph/cluster2-client.admin.78903.log.gz 2023-12-10T23:40:12.718 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.40018.log 2023-12-10T23:40:12.718 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69247.log 2023-12-10T23:40:12.718 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.74303.log: /var/log/ceph/cluster2-client.mirror.2.40018.log: 87.1% -- replaced with /var/log/ceph/cluster2-client.mirror.74303.log.gz 2023-12-10T23:40:12.719 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57331.log 2023-12-10T23:40:12.720 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.69247.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.69247.log.gz 2023-12-10T23:40:12.720 INFO:teuthology.orchestra.run.smithi076.stderr: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.2.40018.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65386.log 2023-12-10T23:40:12.720 INFO:teuthology.orchestra.run.smithi076.stderr: 2023-12-10T23:40:12.721 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.57331.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57331.log.gz 2023-12-10T23:40:12.722 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75240.log 2023-12-10T23:40:12.722 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58604.log 2023-12-10T23:40:12.723 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.65386.log: /var/log/ceph/cluster1-client.mirror.75240.log: 87.0% 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.75240.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.65386.log.gz 2023-12-10T23:40:12.723 INFO:teuthology.orchestra.run.smithi076.stderr: 2023-12-10T23:40:12.724 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74828.log 2023-12-10T23:40:12.724 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68106.log 2023-12-10T23:40:12.725 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.58604.log: /var/log/ceph/cluster2-client.mirror.74828.log: 87.9% 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.74828.log.gz 2023-12-10T23:40:12.725 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.58604.log.gz 2023-12-10T23:40:12.725 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77348.log 2023-12-10T23:40:12.725 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71605.log 2023-12-10T23:40:12.726 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.68106.log: /var/log/ceph/cluster1-client.mirror.77348.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.77348.log.gz 2023-12-10T23:40:12.726 INFO:teuthology.orchestra.run.smithi076.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.68106.log.gz 2023-12-10T23:40:12.726 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77556.log 2023-12-10T23:40:12.726 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56451.log 2023-12-10T23:40:12.727 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.71605.log: /var/log/ceph/cluster1-client.mirror.77556.log: 86.9% 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71605.log.gz 2023-12-10T23:40:12.728 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.77556.log.gz 2023-12-10T23:40:12.728 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58285.log 2023-12-10T23:40:12.728 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47080.log 2023-12-10T23:40:12.729 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.56451.log: /var/log/ceph/cluster1-client.mirror.58285.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56451.log.gz 2023-12-10T23:40:12.729 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60846.log 2023-12-10T23:40:12.729 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.47080.log: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.58285.log.gz 2023-12-10T23:40:12.734 INFO:teuthology.orchestra.run.smithi076.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.47080.log.gz 2023-12-10T23:40:12.734 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78251.log 2023-12-10T23:40:12.735 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39504.log 2023-12-10T23:40:12.735 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.60846.log: /var/log/ceph/cluster1-client.mirror.78251.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78251.log.gz 2023-12-10T23:40:12.735 INFO:teuthology.orchestra.run.smithi076.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.60846.log.gz 2023-12-10T23:40:12.736 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76797.log 2023-12-10T23:40:12.737 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63763.log 2023-12-10T23:40:12.737 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.admin.39504.log: /var/log/ceph/cluster1-client.mirror.76797.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39504.log.gz 2023-12-10T23:40:12.737 INFO:teuthology.orchestra.run.smithi076.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.76797.log.gz 2023-12-10T23:40:12.737 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54592.log 2023-12-10T23:40:12.738 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.63763.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63763.log.gz 2023-12-10T23:40:12.738 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64979.log 2023-12-10T23:40:12.739 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78403.log 2023-12-10T23:40:12.739 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.54592.log: /var/log/ceph/cluster1-client.mirror.64979.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54592.log.gz 2023-12-10T23:40:12.739 INFO:teuthology.orchestra.run.smithi076.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.64979.log.gz 2023-12-10T23:40:12.739 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63930.log 2023-12-10T23:40:12.740 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65233.log 2023-12-10T23:40:12.740 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.admin.78403.log: /var/log/ceph/cluster1-client.mirror.63930.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.admin.78403.log.gz 2023-12-10T23:40:12.740 INFO:teuthology.orchestra.run.smithi076.stderr: 87.5%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50398.log 2023-12-10T23:40:12.741 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.63930.log.gz 2023-12-10T23:40:12.742 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.65233.log: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.65233.log.gz 2023-12-10T23:40:12.743 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74951.log 2023-12-10T23:40:12.743 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74262.log 2023-12-10T23:40:12.744 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.50398.log: /var/log/ceph/cluster2-client.mirror.74951.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.74951.log.gz 2023-12-10T23:40:12.744 INFO:teuthology.orchestra.run.smithi076.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50398.log.gz 2023-12-10T23:40:12.744 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66719.log 2023-12-10T23:40:12.745 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78505.log 2023-12-10T23:40:12.745 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.74262.log: /var/log/ceph/cluster2-client.mirror.66719.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.74262.log.gz 2023-12-10T23:40:12.746 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68362.log 2023-12-10T23:40:12.746 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.78505.log: 9.2% -- replaced with /var/log/ceph/cluster2-client.mirror.78505.log.gz 2023-12-10T23:40:12.747 INFO:teuthology.orchestra.run.smithi076.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.66719.log.gz 2023-12-10T23:40:12.747 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57648.log 2023-12-10T23:40:12.748 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.68362.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.68362.log.gz 2023-12-10T23:40:12.748 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49884.log 2023-12-10T23:40:12.749 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56388.log 2023-12-10T23:40:12.749 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.57648.log: /var/log/ceph/cluster2-client.mirror.49884.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.57648.log.gz 2023-12-10T23:40:12.750 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53448.log 2023-12-10T23:40:12.750 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.56388.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.56388.log.gz 2023-12-10T23:40:12.751 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46845.log 2023-12-10T23:40:12.751 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.53448.log: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.53448.log.gz 2023-12-10T23:40:12.752 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70750.log 2023-12-10T23:40:12.757 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.46845.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46845.log.gz 2023-12-10T23:40:12.758 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67692.log 2023-12-10T23:40:12.758 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.70750.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70750.log.gz 2023-12-10T23:40:12.758 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75975.log 2023-12-10T23:40:12.759 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.67692.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.67692.log.gz 2023-12-10T23:40:12.760 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49983.log 2023-12-10T23:40:12.760 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.75975.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.75975.log.gz 2023-12-10T23:40:12.761 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62414.log 2023-12-10T23:40:12.761 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.49983.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.49983.log.gz 2023-12-10T23:40:12.762 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55629.log 2023-12-10T23:40:12.762 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.62414.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.62414.log.gz 2023-12-10T23:40:12.763 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54672.log 2023-12-10T23:40:12.763 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.55629.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.55629.log.gz 2023-12-10T23:40:12.764 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51851.log 2023-12-10T23:40:12.764 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.54672.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54672.log.gz 2023-12-10T23:40:12.765 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42807.log 2023-12-10T23:40:12.766 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.51851.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51851.log.gz 2023-12-10T23:40:12.766 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42123.log 2023-12-10T23:40:12.771 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.42807.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.42807.log.gz 2023-12-10T23:40:12.772 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42278.log 2023-12-10T23:40:12.773 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.42123.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.42123.log.gz 2023-12-10T23:40:12.773 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74201.log 2023-12-10T23:40:12.774 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.42278.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.42278.log.gz 2023-12-10T23:40:12.774 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66781.log 2023-12-10T23:40:12.775 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.74201.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.74201.log.gz 2023-12-10T23:40:12.775 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57489.log 2023-12-10T23:40:12.777 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.66781.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.66781.log.gz 2023-12-10T23:40:12.777 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52272.log 2023-12-10T23:40:12.778 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.57489.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.57489.log.gz 2023-12-10T23:40:12.779 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.31170.log 2023-12-10T23:40:12.779 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.52272.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.52272.log.gz 2023-12-10T23:40:12.780 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50551.log 2023-12-10T23:40:12.780 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph.tmp-client.admin.31170.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.31170.log.gz 2023-12-10T23:40:12.780 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70390.log 2023-12-10T23:40:12.781 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.50551.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50551.log.gz 2023-12-10T23:40:12.781 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64266.log 2023-12-10T23:40:12.782 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.70390.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.70390.log.gz 2023-12-10T23:40:12.782 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46099.log 2023-12-10T23:40:12.783 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.64266.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.64266.log.gz 2023-12-10T23:40:12.784 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.44799.log 2023-12-10T23:40:12.784 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.46099.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.46099.log.gz 2023-12-10T23:40:12.785 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.59547.log 2023-12-10T23:40:12.803 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.0.44799.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.0.44799.log.gz 2023-12-10T23:40:12.803 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60938.log 2023-12-10T23:40:12.817 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.2.59547.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54987.log 2023-12-10T23:40:12.818 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.60938.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60938.log.gz 2023-12-10T23:40:12.841 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74556.log 2023-12-10T23:40:12.844 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.54987.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54987.log.gz 2023-12-10T23:40:12.857 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46293.log 2023-12-10T23:40:12.865 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.74556.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46648.log 2023-12-10T23:40:12.866 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.46293.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49138.log.gz 2023-12-10T23:40:12.866 INFO:teuthology.orchestra.run.smithi076.stderr: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.46293.log.gz 2023-12-10T23:40:12.867 INFO:teuthology.orchestra.run.smithi076.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48649.log.gz 2023-12-10T23:40:12.867 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62326.log 2023-12-10T23:40:12.868 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.46648.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61264.log 2023-12-10T23:40:12.868 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.62326.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46648.log.gz 2023-12-10T23:40:12.868 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69019.log 2023-12-10T23:40:12.870 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.61264.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.61264.log.gz 2023-12-10T23:40:12.870 INFO:teuthology.orchestra.run.smithi076.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62326.log.gz 2023-12-10T23:40:12.870 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48297.log 2023-12-10T23:40:12.871 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.69019.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43569.log 2023-12-10T23:40:12.872 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.48297.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.69019.log.gz 2023-12-10T23:40:12.872 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39876.log 2023-12-10T23:40:12.873 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.43569.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.43569.log.gz 2023-12-10T23:40:12.873 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62748.log 2023-12-10T23:40:12.874 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.admin.39876.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39876.log.gz 2023-12-10T23:40:12.874 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67857.log 2023-12-10T23:40:12.876 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.62748.log: 94.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62748.log.gz 2023-12-10T23:40:12.877 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50913.log 2023-12-10T23:40:12.877 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.67857.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.67857.log.gz 2023-12-10T23:40:12.878 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74787.log 2023-12-10T23:40:12.879 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.50913.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50913.log.gz 2023-12-10T23:40:12.879 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53427.log 2023-12-10T23:40:12.880 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.74787.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.74787.log.gz 2023-12-10T23:40:12.880 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51241.log 2023-12-10T23:40:12.881 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.53427.log: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.53427.log.gz 2023-12-10T23:40:12.881 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53551.log 2023-12-10T23:40:12.882 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.51241.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51241.log.gz 2023-12-10T23:40:12.883 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78555.log 2023-12-10T23:40:12.884 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.53551.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.53551.log.gz 2023-12-10T23:40:12.885 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52294.log 2023-12-10T23:40:12.885 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.78555.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.78555.log.gz 2023-12-10T23:40:12.885 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70907.log 2023-12-10T23:40:12.902 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.52294.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73432.log 2023-12-10T23:40:12.902 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.70907.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70907.log.gz 2023-12-10T23:40:12.903 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66595.log 2023-12-10T23:40:12.904 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.73432.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.73432.log.gz 2023-12-10T23:40:12.909 INFO:teuthology.orchestra.run.smithi076.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52878.log.gz 2023-12-10T23:40:12.917 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.34845.log 2023-12-10T23:40:12.919 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.66595.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.66595.log.gz 2023-12-10T23:40:12.920 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71312.log 2023-12-10T23:40:12.920 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.admin.34845.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.34845.log.gz 2023-12-10T23:40:12.926 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77627.log 2023-12-10T23:40:12.926 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.71312.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71312.log.gz 2023-12-10T23:40:12.934 INFO:teuthology.orchestra.run.smithi076.stderr: 94.1% -- replaced with /var/log/ceph/cluster1-client.mirror.2.59547.log.gz 2023-12-10T23:40:12.941 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44914.log 2023-12-10T23:40:12.942 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.77627.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.77627.log.gz 2023-12-10T23:40:12.943 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56718.log 2023-12-10T23:40:12.944 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.44914.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44914.log.gz 2023-12-10T23:40:12.944 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48194.log 2023-12-10T23:40:12.945 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.56718.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.56718.log.gz 2023-12-10T23:40:12.946 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78545.log 2023-12-10T23:40:12.947 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.48194.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.48194.log.gz 2023-12-10T23:40:12.947 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.39909.log 2023-12-10T23:40:12.948 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.admin.78545.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.admin.78545.log.gz 2023-12-10T23:40:12.948 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53384.log 2023-12-10T23:40:12.953 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.1.39909.log: 92.8% -- replaced with /var/log/ceph/cluster1-client.mirror.1.39909.log.gz 2023-12-10T23:40:12.953 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58307.log 2023-12-10T23:40:12.969 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.53384.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65409.log 2023-12-10T23:40:12.973 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.58307.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.58307.log.gz 2023-12-10T23:40:12.994 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73242.log 2023-12-10T23:40:12.994 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.65409.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.65409.log.gz 2023-12-10T23:40:13.006 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57916.log 2023-12-10T23:40:13.011 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.73242.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48539.log.gz 2023-12-10T23:40:13.021 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61136.log 2023-12-10T23:40:13.022 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.57916.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57916.log.gz 2023-12-10T23:40:13.037 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66057.log 2023-12-10T23:40:13.038 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.61136.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61136.log.gz 2023-12-10T23:40:13.040 INFO:teuthology.orchestra.run.smithi076.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49884.log.gz 2023-12-10T23:40:13.048 INFO:teuthology.orchestra.run.smithi076.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51678.log.gz 2023-12-10T23:40:13.049 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49963.log 2023-12-10T23:40:13.050 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74497.log 2023-12-10T23:40:13.050 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.66057.log: /var/log/ceph/cluster2-client.mirror.49963.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.66057.log.gz 2023-12-10T23:40:13.050 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69205.log 2023-12-10T23:40:13.051 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.74497.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.74497.log.gz 2023-12-10T23:40:13.051 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76590.log 2023-12-10T23:40:13.053 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.69205.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.69205.log.gz 2023-12-10T23:40:13.054 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75954.log 2023-12-10T23:40:13.055 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.76590.log: 87.1% -- replaced with /var/log/ceph/cluster2-client.mirror.76590.log.gz 2023-12-10T23:40:13.055 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78272.log 2023-12-10T23:40:13.056 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.75954.log: 87.1% -- replaced with /var/log/ceph/cluster2-client.mirror.75954.log.gz 2023-12-10T23:40:13.056 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45049.log 2023-12-10T23:40:13.057 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.78272.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78272.log.gz 2023-12-10T23:40:13.057 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63058.log 2023-12-10T23:40:13.058 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.45049.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.45049.log.gz 2023-12-10T23:40:13.059 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42080.log 2023-12-10T23:40:13.059 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.63058.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.63058.log.gz 2023-12-10T23:40:13.060 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73825.log 2023-12-10T23:40:13.061 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.42080.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.42080.log.gz 2023-12-10T23:40:13.062 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39826.log 2023-12-10T23:40:13.063 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.73825.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.73825.log.gz 2023-12-10T23:40:13.063 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56161.log 2023-12-10T23:40:13.064 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.admin.39826.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39826.log.gz 2023-12-10T23:40:13.064 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49474.log 2023-12-10T23:40:13.066 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.56161.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.56161.log.gz 2023-12-10T23:40:13.066 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69490.log 2023-12-10T23:40:13.067 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.49474.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49474.log.gz 2023-12-10T23:40:13.067 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39132.log 2023-12-10T23:40:13.068 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.69490.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69490.log.gz 2023-12-10T23:40:13.069 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78382.log 2023-12-10T23:40:13.069 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.admin.39132.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39132.log.gz 2023-12-10T23:40:13.070 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45347.log 2023-12-10T23:40:13.070 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.78382.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78382.log.gz 2023-12-10T23:40:13.071 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45177.log 2023-12-10T23:40:13.071 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.45347.log: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.45347.log.gz 2023-12-10T23:40:13.072 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41980.log 2023-12-10T23:40:13.086 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.45177.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40965.log 2023-12-10T23:40:13.086 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.admin.41980.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.41980.log.gz 2023-12-10T23:40:13.093 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40323.log 2023-12-10T23:40:13.094 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.40965.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.40965.log.gz 2023-12-10T23:40:13.108 INFO:teuthology.orchestra.run.smithi076.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53384.log.gz 2023-12-10T23:40:13.108 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52373.log 2023-12-10T23:40:13.109 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.40323.log: 4.7% -- replaced with /var/log/ceph/cluster1-client.mirror.40323.log.gz 2023-12-10T23:40:13.109 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44564.log 2023-12-10T23:40:13.125 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.52373.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68404.log 2023-12-10T23:40:13.131 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.44564.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.44564.log.gz 2023-12-10T23:40:13.137 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44177.log 2023-12-10T23:40:13.138 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.68404.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.68404.log.gz 2023-12-10T23:40:13.153 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49079.log 2023-12-10T23:40:13.159 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.44177.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.44177.log.gz 2023-12-10T23:40:13.173 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68916.log 2023-12-10T23:40:13.174 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.49079.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49079.log.gz 2023-12-10T23:40:13.182 INFO:teuthology.orchestra.run.smithi076.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.73242.log.gz 2023-12-10T23:40:13.186 INFO:teuthology.orchestra.run.smithi076.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48297.log.gz 2023-12-10T23:40:13.190 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66905.log 2023-12-10T23:40:13.190 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.68916.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48669.log 2023-12-10T23:40:13.190 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.66905.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.68916.log.gz 2023-12-10T23:40:13.191 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74118.log 2023-12-10T23:40:13.192 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.48669.log: 79.8% 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.66905.log.gz 2023-12-10T23:40:13.192 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.48669.log.gz 2023-12-10T23:40:13.192 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78432.log 2023-12-10T23:40:13.193 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.74118.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77832.log 2023-12-10T23:40:13.193 INFO:teuthology.orchestra.run.smithi076.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.74118.log.gz/var/log/ceph/cluster2-client.admin.78432.log: 2023-12-10T23:40:13.194 INFO:teuthology.orchestra.run.smithi076.stderr: 7.1% -- replaced with /var/log/ceph/cluster2-client.admin.78432.log.gz 2023-12-10T23:40:13.194 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44024.log 2023-12-10T23:40:13.194 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.77832.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43156.log 2023-12-10T23:40:13.194 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.77832.log.gz 2023-12-10T23:40:13.195 INFO:teuthology.orchestra.run.smithi076.stderr:gzip/var/log/ceph/cluster1-client.mirror.44024.log: -5 --verbose -- /var/log/ceph/cluster1-client.admin.42896.log 2023-12-10T23:40:13.195 INFO:teuthology.orchestra.run.smithi076.stderr: 88.7%/var/log/ceph/cluster2-client.mirror.43156.log: -- replaced with /var/log/ceph/cluster1-client.mirror.44024.log.gz 2023-12-10T23:40:13.195 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40306.log 2023-12-10T23:40:13.197 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.admin.42896.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.42896.log.gz 2023-12-10T23:40:13.197 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64643.log 2023-12-10T23:40:13.197 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.40306.log: 4.7% -- replaced with /var/log/ceph/cluster1-client.mirror.40306.log.gz 2023-12-10T23:40:13.198 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76299.log 2023-12-10T23:40:13.198 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.64643.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.64643.log.gz 2023-12-10T23:40:13.199 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71470.log 2023-12-10T23:40:13.200 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.76299.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.76299.log.gz 2023-12-10T23:40:13.200 INFO:teuthology.orchestra.run.smithi076.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43156.log.gz 2023-12-10T23:40:13.200 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.42493.log 2023-12-10T23:40:13.200 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.71470.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46406.log 2023-12-10T23:40:13.201 INFO:teuthology.orchestra.run.smithi076.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.71470.log.gz/var/log/ceph/cluster2-client.mirror.2.42493.log: 2023-12-10T23:40:13.201 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53707.log 2023-12-10T23:40:13.206 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.46406.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.2.42493.log.gz 2023-12-10T23:40:13.206 INFO:teuthology.orchestra.run.smithi076.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46406.log.gz 2023-12-10T23:40:13.207 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.43843.log 2023-12-10T23:40:13.207 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68660.log 2023-12-10T23:40:13.208 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.53707.log: /var/log/ceph/cluster1-client.mirror.1.43843.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.53707.log.gz 2023-12-10T23:40:13.209 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51814.log 2023-12-10T23:40:13.211 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.68660.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.68660.log.gz 2023-12-10T23:40:13.212 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59332.log 2023-12-10T23:40:13.218 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.51814.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.74556.log.gz 2023-12-10T23:40:13.220 INFO:teuthology.orchestra.run.smithi076.stderr: 93.1% -- replaced with /var/log/ceph/cluster1-client.mirror.1.43843.log.gz 2023-12-10T23:40:13.221 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65495.log 2023-12-10T23:40:13.221 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55187.log 2023-12-10T23:40:13.222 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.59332.log: /var/log/ceph/cluster2-client.mirror.65495.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59332.log.gz 2023-12-10T23:40:13.222 INFO:teuthology.orchestra.run.smithi076.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65495.log.gz 2023-12-10T23:40:13.222 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50418.log 2023-12-10T23:40:13.223 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.55187.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.55187.log.gz 2023-12-10T23:40:13.223 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52252.log 2023-12-10T23:40:13.224 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56738.log 2023-12-10T23:40:13.224 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.50418.log: /var/log/ceph/cluster2-client.mirror.52252.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50418.log.gz 2023-12-10T23:40:13.225 INFO:teuthology.orchestra.run.smithi076.stderr: 88.7%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53943.log 2023-12-10T23:40:13.225 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.52252.log.gz 2023-12-10T23:40:13.225 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.56738.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56738.log.gz 2023-12-10T23:40:13.226 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.77719.log 2023-12-10T23:40:13.226 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43043.log 2023-12-10T23:40:13.227 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.53943.log: /var/log/ceph/cluster1-client.admin.77719.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.77719.log.gz 2023-12-10T23:40:13.228 INFO:teuthology.orchestra.run.smithi076.stderr: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.53943.log.gz 2023-12-10T23:40:13.228 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67898.log 2023-12-10T23:40:13.229 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71492.log 2023-12-10T23:40:13.229 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.43043.log: /var/log/ceph/cluster2-client.mirror.67898.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.43043.log.gz 2023-12-10T23:40:13.230 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49226.log 2023-12-10T23:40:13.230 INFO:teuthology.orchestra.run.smithi076.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52294.log.gz 2023-12-10T23:40:13.231 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.71492.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.71492.log.gz 2023-12-10T23:40:13.231 INFO:teuthology.orchestra.run.smithi076.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.67898.log.gz 2023-12-10T23:40:13.231 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55854.log 2023-12-10T23:40:13.231 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64220.log 2023-12-10T23:40:13.231 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.49226.log: /var/log/ceph/cluster1-client.mirror.55854.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67547.log 2023-12-10T23:40:13.232 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.64220.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.55854.log.gz 2023-12-10T23:40:13.232 INFO:teuthology.orchestra.run.smithi076.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.64220.log.gz 2023-12-10T23:40:13.242 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60753.log 2023-12-10T23:40:13.242 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.67547.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65297.log 2023-12-10T23:40:13.243 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.60753.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.67547.log.gz 2023-12-10T23:40:13.243 INFO:teuthology.orchestra.run.smithi076.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.60753.log.gz 2023-12-10T23:40:13.243 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65213.log 2023-12-10T23:40:13.244 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.65297.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58457.log 2023-12-10T23:40:13.244 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.65213.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.65297.log.gz 2023-12-10T23:40:13.245 INFO:teuthology.orchestra.run.smithi076.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65213.log.gz 2023-12-10T23:40:13.246 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40272.log 2023-12-10T23:40:13.247 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49184.log 2023-12-10T23:40:13.247 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.58457.log: /var/log/ceph/cluster1-client.admin.40272.log: 66.9% -- replaced with /var/log/ceph/cluster1-client.admin.40272.log.gz 2023-12-10T23:40:13.247 INFO:teuthology.orchestra.run.smithi076.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.58457.log.gz 2023-12-10T23:40:13.247 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73867.log 2023-12-10T23:40:13.248 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47189.log 2023-12-10T23:40:13.248 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.49184.log: /var/log/ceph/cluster2-client.mirror.73867.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49184.log.gz 2023-12-10T23:40:13.249 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48238.log 2023-12-10T23:40:13.254 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.47189.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47189.log.gz 2023-12-10T23:40:13.254 INFO:teuthology.orchestra.run.smithi076.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.45177.log.gz 2023-12-10T23:40:13.254 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61354.log 2023-12-10T23:40:13.255 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62437.log 2023-12-10T23:40:13.255 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.48238.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.48238.log.gz 2023-12-10T23:40:13.255 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.61354.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71155.log 2023-12-10T23:40:13.256 INFO:teuthology.orchestra.run.smithi076.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61354.log.gz 2023-12-10T23:40:13.257 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.62437.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76862.log 2023-12-10T23:40:13.257 INFO:teuthology.orchestra.run.smithi076.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.62437.log.gz 2023-12-10T23:40:13.257 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.71155.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72866.log 2023-12-10T23:40:13.257 INFO:teuthology.orchestra.run.smithi076.stderr: 86.8%/var/log/ceph/cluster2-client.mirror.76862.log: -- replaced with /var/log/ceph/cluster1-client.mirror.71155.log.gz 2023-12-10T23:40:13.258 INFO:teuthology.orchestra.run.smithi076.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.76862.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61621.log 2023-12-10T23:40:13.258 INFO:teuthology.orchestra.run.smithi076.stderr: 2023-12-10T23:40:13.260 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58898.log 2023-12-10T23:40:13.260 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.72866.log: /var/log/ceph/cluster1-client.mirror.61621.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.72866.log.gz 2023-12-10T23:40:13.260 INFO:teuthology.orchestra.run.smithi076.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.61621.log.gz 2023-12-10T23:40:13.260 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76341.log 2023-12-10T23:40:13.260 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.58898.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62239.log 2023-12-10T23:40:13.260 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.76341.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.58898.log.gz 2023-12-10T23:40:13.260 INFO:teuthology.orchestra.run.smithi076.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.76341.log.gz 2023-12-10T23:40:13.260 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65928.log 2023-12-10T23:40:13.262 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.62239.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68957.log 2023-12-10T23:40:13.262 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.65928.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.62239.log.gz 2023-12-10T23:40:13.262 INFO:teuthology.orchestra.run.smithi076.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.65928.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46822.log 2023-12-10T23:40:13.262 INFO:teuthology.orchestra.run.smithi076.stderr: 2023-12-10T23:40:13.262 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.68957.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59915.log 2023-12-10T23:40:13.263 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.46822.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.68957.log.gz 2023-12-10T23:40:13.263 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44388.log 2023-12-10T23:40:13.264 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.59915.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.59915.log.gz 2023-12-10T23:40:13.264 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53305.log 2023-12-10T23:40:13.265 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.44388.log: 88.8% -- replaced with /var/log/ceph/cluster1-client.mirror.44388.log.gz 2023-12-10T23:40:13.266 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52935.log 2023-12-10T23:40:13.267 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.53305.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.46822.log.gz 2023-12-10T23:40:13.268 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55716.log 2023-12-10T23:40:13.268 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.52935.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52935.log.gz 2023-12-10T23:40:13.269 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54523.log 2023-12-10T23:40:13.271 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.55716.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.55716.log.gz 2023-12-10T23:40:13.271 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64445.log 2023-12-10T23:40:13.272 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.54523.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54523.log.gz 2023-12-10T23:40:13.272 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47447.log 2023-12-10T23:40:13.273 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.64445.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.64445.log.gz 2023-12-10T23:40:13.274 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64735.log 2023-12-10T23:40:13.280 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.47447.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47447.log.gz 2023-12-10T23:40:13.280 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44134.log 2023-12-10T23:40:13.281 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.64735.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.64735.log.gz 2023-12-10T23:40:13.281 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54729.log 2023-12-10T23:40:13.287 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.44134.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.44134.log.gz 2023-12-10T23:40:13.287 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47676.log 2023-12-10T23:40:13.288 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.54729.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54729.log.gz 2023-12-10T23:40:13.288 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76944.log 2023-12-10T23:40:13.289 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.47676.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.47676.log.gz 2023-12-10T23:40:13.290 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40909.log 2023-12-10T23:40:13.290 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.76944.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.76944.log.gz 2023-12-10T23:40:13.291 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62193.log 2023-12-10T23:40:13.291 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.admin.40909.log: 46.6% -- replaced with /var/log/ceph/cluster2-client.admin.40909.log.gz 2023-12-10T23:40:13.292 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58175.log 2023-12-10T23:40:13.293 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.62193.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.62193.log.gz 2023-12-10T23:40:13.293 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.41203.log 2023-12-10T23:40:13.294 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.58175.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58175.log.gz 2023-12-10T23:40:13.294 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75679.log 2023-12-10T23:40:13.304 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.1.41203.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49963.log.gz 2023-12-10T23:40:13.309 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48929.log 2023-12-10T23:40:13.310 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.75679.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.75679.log.gz 2023-12-10T23:40:13.313 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39905.log 2023-12-10T23:40:13.313 INFO:teuthology.orchestra.run.smithi076.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.1.41203.log.gz 2023-12-10T23:40:13.313 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.48929.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50440.log 2023-12-10T23:40:13.313 INFO:teuthology.orchestra.run.smithi076.stderr: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48929.log.gz 2023-12-10T23:40:13.313 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.39905.log: 78.7% -- replaced with /var/log/ceph/cluster2-client.mirror.39905.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45113.log 2023-12-10T23:40:13.313 INFO:teuthology.orchestra.run.smithi076.stderr: 2023-12-10T23:40:13.330 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.50440.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35065.log 2023-12-10T23:40:13.335 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.45113.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45113.log.gz 2023-12-10T23:40:13.336 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45090.log 2023-12-10T23:40:13.336 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.admin.35065.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35065.log.gz 2023-12-10T23:40:13.337 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66203.log 2023-12-10T23:40:13.342 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.45090.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.45090.log.gz 2023-12-10T23:40:13.343 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75659.log 2023-12-10T23:40:13.344 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.66203.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.66203.log.gz 2023-12-10T23:40:13.344 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72648.log 2023-12-10T23:40:13.345 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.75659.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.75659.log.gz 2023-12-10T23:40:13.345 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78124.log 2023-12-10T23:40:13.346 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.72648.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.72648.log.gz 2023-12-10T23:40:13.347 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66409.log 2023-12-10T23:40:13.347 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.78124.log: 10.2% -- replaced with /var/log/ceph/cluster1-client.mirror.78124.log.gz 2023-12-10T23:40:13.348 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50856.log 2023-12-10T23:40:13.349 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.66409.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.66409.log.gz 2023-12-10T23:40:13.350 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74766.log 2023-12-10T23:40:13.365 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.50856.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64600.log 2023-12-10T23:40:13.366 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.74766.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.74766.log.gz 2023-12-10T23:40:13.367 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44047.log 2023-12-10T23:40:13.368 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.64600.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.64600.log.gz 2023-12-10T23:40:13.379 INFO:teuthology.orchestra.run.smithi076.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51814.log.gz 2023-12-10T23:40:13.381 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54184.log 2023-12-10T23:40:13.382 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.44047.log: 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.44047.log.gz 2023-12-10T23:40:13.383 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45480.log 2023-12-10T23:40:13.385 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.54184.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54184.log.gz 2023-12-10T23:40:13.385 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58921.log 2023-12-10T23:40:13.391 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.45480.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.45480.log.gz 2023-12-10T23:40:13.391 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66120.log 2023-12-10T23:40:13.392 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.58921.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58921.log.gz 2023-12-10T23:40:13.392 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70412.log 2023-12-10T23:40:13.393 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.66120.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.66120.log.gz 2023-12-10T23:40:13.394 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67091.log 2023-12-10T23:40:13.395 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.70412.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70412.log.gz 2023-12-10T23:40:13.395 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69081.log 2023-12-10T23:40:13.397 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.67091.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.67091.log.gz 2023-12-10T23:40:13.397 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46057.log 2023-12-10T23:40:13.399 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.69081.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.69081.log.gz 2023-12-10T23:40:13.399 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75428.log 2023-12-10T23:40:13.405 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.46057.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46057.log.gz 2023-12-10T23:40:13.405 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59378.log 2023-12-10T23:40:13.406 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.75428.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.75428.log.gz 2023-12-10T23:40:13.407 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52856.log 2023-12-10T23:40:13.408 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.59378.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.59378.log.gz 2023-12-10T23:40:13.408 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60667.log 2023-12-10T23:40:13.409 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.52856.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52856.log.gz 2023-12-10T23:40:13.409 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46716.log 2023-12-10T23:40:13.410 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.60667.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.60667.log.gz 2023-12-10T23:40:13.411 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54206.log 2023-12-10T23:40:13.416 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.46716.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46716.log.gz 2023-12-10T23:40:13.417 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78590.log 2023-12-10T23:40:13.418 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.54206.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54206.log.gz 2023-12-10T23:40:13.418 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73746.log 2023-12-10T23:40:13.419 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.78590.log: 9.2% -- replaced with /var/log/ceph/cluster2-client.mirror.78590.log.gz 2023-12-10T23:40:13.419 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77145.log 2023-12-10T23:40:13.420 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.73746.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.73746.log.gz 2023-12-10T23:40:13.420 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59229.log 2023-12-10T23:40:13.421 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.77145.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.77145.log.gz 2023-12-10T23:40:13.422 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48019.log 2023-12-10T23:40:13.424 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.59229.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.59229.log.gz 2023-12-10T23:40:13.424 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44753.log 2023-12-10T23:40:13.425 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.48019.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.48019.log.gz 2023-12-10T23:40:13.425 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46449.log 2023-12-10T23:40:13.426 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.44753.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.44753.log.gz 2023-12-10T23:40:13.427 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66368.log 2023-12-10T23:40:13.427 INFO:teuthology.orchestra.run.smithi076.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52373.log.gz/var/log/ceph/cluster2-client.mirror.46449.log: 2023-12-10T23:40:13.432 INFO:teuthology.orchestra.run.smithi076.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46449.log.gz 2023-12-10T23:40:13.433 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74534.log 2023-12-10T23:40:13.433 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49397.log 2023-12-10T23:40:13.434 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.66368.log: /var/log/ceph/cluster2-client.mirror.74534.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.66368.log.gz 2023-12-10T23:40:13.434 INFO:teuthology.orchestra.run.smithi076.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.74534.log.gz 2023-12-10T23:40:13.434 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50072.log 2023-12-10T23:40:13.435 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68999.log 2023-12-10T23:40:13.435 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.49397.log: /var/log/ceph/cluster2-client.mirror.50072.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50072.log.gz 2023-12-10T23:40:13.435 INFO:teuthology.orchestra.run.smithi076.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49397.log.gz 2023-12-10T23:40:13.436 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73520.log 2023-12-10T23:40:13.437 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56181.log 2023-12-10T23:40:13.437 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.68999.log: /var/log/ceph/cluster1-client.mirror.73520.log: 86.9% 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.68999.log.gz 2023-12-10T23:40:13.437 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.73520.log.gz 2023-12-10T23:40:13.437 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62631.log 2023-12-10T23:40:13.437 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60128.log 2023-12-10T23:40:13.438 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.56181.log: /var/log/ceph/cluster2-client.mirror.62631.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.62631.log.gz 2023-12-10T23:40:13.439 INFO:teuthology.orchestra.run.smithi076.stderr: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.56181.log.gz 2023-12-10T23:40:13.439 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74890.log 2023-12-10T23:40:13.440 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58627.log 2023-12-10T23:40:13.440 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.60128.log: /var/log/ceph/cluster2-client.mirror.74890.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60128.log.gz 2023-12-10T23:40:13.440 INFO:teuthology.orchestra.run.smithi076.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.74890.log.gz 2023-12-10T23:40:13.441 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74095.log 2023-12-10T23:40:13.442 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.39905.log 2023-12-10T23:40:13.442 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.58627.log: /var/log/ceph/cluster1-client.mirror.74095.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.74095.log.gz 2023-12-10T23:40:13.442 INFO:teuthology.orchestra.run.smithi076.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.58627.log.gz 2023-12-10T23:40:13.442 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69061.log 2023-12-10T23:40:13.443 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69647.log 2023-12-10T23:40:13.443 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.0.39905.log: /var/log/ceph/cluster2-client.mirror.69061.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.69061.log.gz 2023-12-10T23:40:13.445 INFO:teuthology.orchestra.run.smithi076.stderr: 91.3% -- replaced with /var/log/ceph/cluster1-client.mirror.0.39905.log.gz 2023-12-10T23:40:13.445 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77604.log 2023-12-10T23:40:13.446 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67960.log 2023-12-10T23:40:13.447 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.69647.log: /var/log/ceph/cluster1-client.mirror.77604.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.69647.log.gz 2023-12-10T23:40:13.447 INFO:teuthology.orchestra.run.smithi076.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.77604.log.gz 2023-12-10T23:40:13.447 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67133.log 2023-12-10T23:40:13.447 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76036.log 2023-12-10T23:40:13.448 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.67960.log: /var/log/ceph/cluster2-client.mirror.67133.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.67133.log.gz 2023-12-10T23:40:13.448 INFO:teuthology.orchestra.run.smithi076.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.67960.log.gz 2023-12-10T23:40:13.448 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73142.log 2023-12-10T23:40:13.449 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75172.log 2023-12-10T23:40:13.450 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.76036.log: /var/log/ceph/cluster2-client.mirror.73142.log: 87.9% 87.1% -- replaced with /var/log/ceph/cluster2-client.mirror.76036.log.gz 2023-12-10T23:40:13.450 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.73142.log.gz 2023-12-10T23:40:13.450 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45026.log 2023-12-10T23:40:13.451 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55788.log 2023-12-10T23:40:13.451 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.75172.log: /var/log/ceph/cluster1-client.mirror.45026.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.75172.log.gz 2023-12-10T23:40:13.451 INFO:teuthology.orchestra.run.smithi076.stderr: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45026.log.gz 2023-12-10T23:40:13.451 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42215.log 2023-12-10T23:40:13.452 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44934.log 2023-12-10T23:40:13.453 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.55788.log: /var/log/ceph/cluster1-client.mirror.42215.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.42215.log.gz 2023-12-10T23:40:13.453 INFO:teuthology.orchestra.run.smithi076.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.55788.log.gz 2023-12-10T23:40:13.453 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42720.log 2023-12-10T23:40:13.454 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59119.log 2023-12-10T23:40:13.454 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.44934.log: /var/log/ceph/cluster1-client.mirror.42720.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.44934.log.gz 2023-12-10T23:40:13.454 INFO:teuthology.orchestra.run.smithi076.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.42720.log.gz 2023-12-10T23:40:13.454 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58132.log 2023-12-10T23:40:13.455 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77108.log 2023-12-10T23:40:13.455 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.58132.log: /var/log/ceph/cluster1-client.mirror.59119.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59119.log.gz 2023-12-10T23:40:13.456 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56296.log 2023-12-10T23:40:13.456 INFO:teuthology.orchestra.run.smithi076.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.58132.log.gz 2023-12-10T23:40:13.456 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.77108.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.77108.log.gz 2023-12-10T23:40:13.457 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73962.log 2023-12-10T23:40:13.457 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62700.log 2023-12-10T23:40:13.458 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.56296.log: /var/log/ceph/cluster1-client.mirror.73962.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.56296.log.gz 2023-12-10T23:40:13.458 INFO:teuthology.orchestra.run.smithi076.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.73962.log.gz 2023-12-10T23:40:13.458 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78782.log 2023-12-10T23:40:13.459 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52500.log 2023-12-10T23:40:13.459 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.62700.log: /var/log/ceph/cluster2-client.mirror.78782.log: 10.2% -- replaced with /var/log/ceph/cluster2-client.mirror.78782.log.gz 2023-12-10T23:40:13.459 INFO:teuthology.orchestra.run.smithi076.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.62700.log.gz 2023-12-10T23:40:13.459 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.73012.log 2023-12-10T23:40:13.460 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69940.log 2023-12-10T23:40:13.460 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.52500.log: /var/log/ceph/cluster2-client.mirror.3.73012.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52500.log.gz 2023-12-10T23:40:13.460 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78752.log 2023-12-10T23:40:13.461 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.69940.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.69940.log.gz 2023-12-10T23:40:13.462 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67423.log 2023-12-10T23:40:13.462 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.admin.78752.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.admin.78752.log.gz 2023-12-10T23:40:13.462 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39851.log 2023-12-10T23:40:13.463 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.67423.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.67423.log.gz 2023-12-10T23:40:13.463 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43112.log 2023-12-10T23:40:13.464 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.admin.39851.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39851.log.gz 2023-12-10T23:40:13.464 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56005.log 2023-12-10T23:40:13.465 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.43112.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43112.log.gz 2023-12-10T23:40:13.465 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71020.log 2023-12-10T23:40:13.466 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.56005.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.56005.log.gz 2023-12-10T23:40:13.466 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61067.log 2023-12-10T23:40:13.467 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.71020.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.71020.log.gz 2023-12-10T23:40:13.467 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.0.log 2023-12-10T23:40:13.468 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.61067.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.61067.log.gz 2023-12-10T23:40:13.468 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60414.log 2023-12-10T23:40:13.481 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-osd.0.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.44608.log 2023-12-10T23:40:13.482 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.60414.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.60414.log.gz 2023-12-10T23:40:13.500 INFO:teuthology.orchestra.run.smithi076.stderr: 95.3% -- replaced with /var/log/ceph/cluster2-client.mirror.3.73012.log.gz 2023-12-10T23:40:13.501 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70862.log 2023-12-10T23:40:13.503 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.0.44608.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53305.log.gz 2023-12-10T23:40:13.504 INFO:teuthology.orchestra.run.smithi076.stderr: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.0.44608.log.gz 2023-12-10T23:40:13.504 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45234.log 2023-12-10T23:40:13.505 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74678.log 2023-12-10T23:40:13.505 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.70862.log: /var/log/ceph/cluster1-client.mirror.45234.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70862.log.gz 2023-12-10T23:40:13.506 INFO:teuthology.orchestra.run.smithi076.stderr: 87.8%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70840.log 2023-12-10T23:40:13.506 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.45234.log.gz 2023-12-10T23:40:13.507 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.74678.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.74678.log.gz 2023-12-10T23:40:13.508 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39496.log 2023-12-10T23:40:13.508 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68043.log 2023-12-10T23:40:13.508 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.70840.log: /var/log/ceph/cluster2-client.admin.39496.log: 0.0% 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70840.log.gz 2023-12-10T23:40:13.508 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.39496.log.gz 2023-12-10T23:40:13.508 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53058.log 2023-12-10T23:40:13.509 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76776.log 2023-12-10T23:40:13.509 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.68043.log: /var/log/ceph/cluster1-client.mirror.53058.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.68043.log.gz 2023-12-10T23:40:13.509 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52622.log 2023-12-10T23:40:13.510 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.76776.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.76776.log.gz 2023-12-10T23:40:13.511 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49842.log 2023-12-10T23:40:13.512 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.52622.log: 95.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53058.log.gz 2023-12-10T23:40:13.514 INFO:teuthology.orchestra.run.smithi076.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50856.log.gz 2023-12-10T23:40:13.522 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44799.log 2023-12-10T23:40:13.522 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.49842.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75325.log 2023-12-10T23:40:13.523 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.44799.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49842.log.gz 2023-12-10T23:40:13.523 INFO:teuthology.orchestra.run.smithi076.stderr: 94.1% -- replaced with /var/log/ceph/cluster1-client.mirror.44799.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60345.log 2023-12-10T23:40:13.523 INFO:teuthology.orchestra.run.smithi076.stderr: 2023-12-10T23:40:13.524 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.75325.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57028.log 2023-12-10T23:40:13.525 INFO:teuthology.orchestra.run.smithi076.stderr: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.75325.log.gz/var/log/ceph/cluster1-client.mirror.60345.log: 2023-12-10T23:40:13.525 INFO:teuthology.orchestra.run.smithi076.stderr: 87.4%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57512.log 2023-12-10T23:40:13.525 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.60345.log.gz 2023-12-10T23:40:13.526 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.57028.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64689.log 2023-12-10T23:40:13.527 INFO:teuthology.orchestra.run.smithi076.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57028.log.gz 2023-12-10T23:40:13.527 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.57512.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59938.log 2023-12-10T23:40:13.527 INFO:teuthology.orchestra.run.smithi076.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.57512.log.gz/var/log/ceph/cluster1-client.mirror.64689.log: 2023-12-10T23:40:13.527 INFO:teuthology.orchestra.run.smithi076.stderr: 87.6%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63976.log 2023-12-10T23:40:13.527 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.64689.log.gz 2023-12-10T23:40:13.528 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.59938.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67195.log 2023-12-10T23:40:13.528 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.63976.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.59938.log.gz 2023-12-10T23:40:13.529 INFO:teuthology.orchestra.run.smithi076.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.63976.log.gz 2023-12-10T23:40:13.529 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51792.log 2023-12-10T23:40:13.529 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73200.log 2023-12-10T23:40:13.530 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.67195.log: /var/log/ceph/cluster2-client.mirror.51792.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.67195.log.gz 2023-12-10T23:40:13.530 INFO:teuthology.orchestra.run.smithi076.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51792.log.gz 2023-12-10T23:40:13.530 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76077.log 2023-12-10T23:40:13.531 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.73200.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68424.log 2023-12-10T23:40:13.531 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.76077.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.73200.log.gz 2023-12-10T23:40:13.531 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57576.log 2023-12-10T23:40:13.531 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.68424.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.76077.log.gz 2023-12-10T23:40:13.532 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53469.log 2023-12-10T23:40:13.532 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.57576.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.68424.log.gz 2023-12-10T23:40:13.534 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46339.log 2023-12-10T23:40:13.534 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.53469.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53469.log.gz 2023-12-10T23:40:13.534 INFO:teuthology.orchestra.run.smithi076.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.57576.log.gz 2023-12-10T23:40:13.534 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67506.log 2023-12-10T23:40:13.535 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.46339.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54379.log 2023-12-10T23:40:13.536 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.67506.log: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.46339.log.gz 2023-12-10T23:40:13.536 INFO:teuthology.orchestra.run.smithi076.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.67506.log.gz 2023-12-10T23:40:13.536 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50568.log 2023-12-10T23:40:13.536 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.54379.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47553.log 2023-12-10T23:40:13.537 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.50568.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.54379.log.gz 2023-12-10T23:40:13.537 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59165.log 2023-12-10T23:40:13.537 INFO:teuthology.orchestra.run.smithi076.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50568.log.gz/var/log/ceph/cluster1-client.mirror.47553.log: 2023-12-10T23:40:13.538 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77980.log 2023-12-10T23:40:13.538 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.59165.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.59165.log.gz 2023-12-10T23:40:13.538 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75387.log 2023-12-10T23:40:13.539 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.77980.log: 49.4% -- replaced with /var/log/ceph/cluster1-client.mirror.77980.log.gz 2023-12-10T23:40:13.539 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78483.log 2023-12-10T23:40:13.540 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.75387.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.75387.log.gz 2023-12-10T23:40:13.541 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51834.log 2023-12-10T23:40:13.543 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.78483.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.78483.log.gz 2023-12-10T23:40:13.543 INFO:teuthology.orchestra.run.smithi076.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49226.log.gz 2023-12-10T23:40:13.543 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62262.log 2023-12-10T23:40:13.544 INFO:teuthology.orchestra.run.smithi076.stderr: 93.5%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64063.log 2023-12-10T23:40:13.544 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.47553.log.gz 2023-12-10T23:40:13.544 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.51834.log: /var/log/ceph/cluster1-client.mirror.62262.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51834.log.gz 2023-12-10T23:40:13.544 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52799.log 2023-12-10T23:40:13.544 INFO:teuthology.orchestra.run.smithi076.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.62262.log.gz 2023-12-10T23:40:13.544 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.64063.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39268.log 2023-12-10T23:40:13.545 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77125.log 2023-12-10T23:40:13.545 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.52799.log: /var/log/ceph/cluster2-client.admin.39268.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39268.log.gz 2023-12-10T23:40:13.545 INFO:teuthology.orchestra.run.smithi076.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.64063.log.gz 2023-12-10T23:40:13.546 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.51115.log 2023-12-10T23:40:13.546 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.77125.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66244.log 2023-12-10T23:40:13.547 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.0.51115.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.77125.log.gz 2023-12-10T23:40:13.547 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58875.log 2023-12-10T23:40:13.547 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.66244.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.66244.log.gz 2023-12-10T23:40:13.548 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74180.log 2023-12-10T23:40:13.548 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.58875.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.58875.log.gz 2023-12-10T23:40:13.549 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40357.log 2023-12-10T23:40:13.550 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.74180.log: 87.1% -- replaced with /var/log/ceph/cluster2-client.mirror.74180.log.gz 2023-12-10T23:40:13.550 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60151.log 2023-12-10T23:40:13.551 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.40357.log: 57.1% -- replaced with /var/log/ceph/cluster1-client.mirror.40357.log.gz 2023-12-10T23:40:13.551 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50109.log 2023-12-10T23:40:13.552 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.60151.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.60151.log.gz 2023-12-10T23:40:13.552 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66016.log 2023-12-10T23:40:13.553 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.50109.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50109.log.gz 2023-12-10T23:40:13.553 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44521.log 2023-12-10T23:40:13.554 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.66016.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.66016.log.gz 2023-12-10T23:40:13.555 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65255.log 2023-12-10T23:40:13.560 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.44521.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44521.log.gz 2023-12-10T23:40:13.560 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57241.log 2023-12-10T23:40:13.561 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.65255.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65255.log.gz 2023-12-10T23:40:13.562 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63740.log 2023-12-10T23:40:13.562 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.57241.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.57241.log.gz 2023-12-10T23:40:13.563 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60435.log 2023-12-10T23:40:13.564 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.63740.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.63740.log.gz 2023-12-10T23:40:13.564 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66141.log 2023-12-10T23:40:13.566 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.60435.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.60435.log.gz 2023-12-10T23:40:13.566 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61685.log 2023-12-10T23:40:13.567 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.66141.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.66141.log.gz 2023-12-10T23:40:13.568 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54500.log 2023-12-10T23:40:13.570 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.61685.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61685.log.gz 2023-12-10T23:40:13.570 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48966.log 2023-12-10T23:40:13.571 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.54500.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.54500.log.gz 2023-12-10T23:40:13.571 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56493.log 2023-12-10T23:40:13.572 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.48966.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48966.log.gz 2023-12-10T23:40:13.572 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39914.log 2023-12-10T23:40:13.573 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.56493.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.56493.log.gz 2023-12-10T23:40:13.574 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62131.log 2023-12-10T23:40:13.582 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.39914.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.73867.log.gz 2023-12-10T23:40:13.589 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45743.log 2023-12-10T23:40:13.591 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.62131.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.62131.log.gz 2023-12-10T23:40:13.591 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59355.log 2023-12-10T23:40:13.593 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.45743.log: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.39914.log.gz 2023-12-10T23:40:13.598 INFO:teuthology.orchestra.run.smithi076.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.45743.log.gz 2023-12-10T23:40:13.599 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58521.log 2023-12-10T23:40:13.599 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56830.log 2023-12-10T23:40:13.600 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.59355.log: /var/log/ceph/cluster1-client.mirror.58521.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59355.log.gz 2023-12-10T23:40:13.600 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56959.log 2023-12-10T23:40:13.601 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.56830.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.56830.log.gz 2023-12-10T23:40:13.601 INFO:teuthology.orchestra.run.smithi076.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.58521.log.gz 2023-12-10T23:40:13.602 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46469.log 2023-12-10T23:40:13.602 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46150.log 2023-12-10T23:40:13.603 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.56959.log: /var/log/ceph/cluster1-client.mirror.46469.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56959.log.gz 2023-12-10T23:40:13.603 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39158.log 2023-12-10T23:40:13.604 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.admin.46150.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.46150.log.gz 2023-12-10T23:40:13.604 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49536.log 2023-12-10T23:40:13.605 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.admin.39158.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39158.log.gz 2023-12-10T23:40:13.605 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63717.log 2023-12-10T23:40:13.607 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.49536.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.46469.log.gz 2023-12-10T23:40:13.617 INFO:teuthology.orchestra.run.smithi076.stderr: 94.2% -- replaced with /var/log/ceph/cluster2-client.mirror.0.51115.log.gz 2023-12-10T23:40:13.622 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41203.log 2023-12-10T23:40:13.622 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.63717.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51320.log 2023-12-10T23:40:13.623 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.41203.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.63717.log.gz 2023-12-10T23:40:13.623 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60001.log 92.3% 2023-12-10T23:40:13.623 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.41203.log.gz 2023-12-10T23:40:13.624 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.51320.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51320.log.gz 2023-12-10T23:40:13.624 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73685.log 2023-12-10T23:40:13.624 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.60001.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68751.log 2023-12-10T23:40:13.625 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.73685.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.60001.log.gz 2023-12-10T23:40:13.626 INFO:teuthology.orchestra.run.smithi076.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.73685.log.gz 2023-12-10T23:40:13.626 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77853.log 2023-12-10T23:40:13.626 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.68751.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56807.log 2023-12-10T23:40:13.627 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.77853.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.68751.log.gz 2023-12-10T23:40:13.627 INFO:teuthology.orchestra.run.smithi076.stderr: 68.5% -- replaced with /var/log/ceph/cluster1-client.mirror.77853.log.gz 2023-12-10T23:40:13.628 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69535.log 2023-12-10T23:40:13.628 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54264.log 2023-12-10T23:40:13.628 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.56807.log: /var/log/ceph/cluster1-client.mirror.69535.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69535.log.gz 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56807.log.gz 2023-12-10T23:40:13.628 INFO:teuthology.orchestra.run.smithi076.stderr: 2023-12-10T23:40:13.629 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64776.log 2023-12-10T23:40:13.629 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.2.log 2023-12-10T23:40:13.629 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.54264.log: /var/log/ceph/cluster1-client.mirror.64776.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.54264.log.gz 2023-12-10T23:40:13.629 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74514.log 2023-12-10T23:40:13.630 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-osd.2.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.64776.log.gz 2023-12-10T23:40:13.641 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.49558.log 2023-12-10T23:40:13.643 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.74514.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.74514.log.gz 2023-12-10T23:40:13.643 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45304.log 2023-12-10T23:40:13.658 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.1.49558.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75766.log 2023-12-10T23:40:13.659 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.45304.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50440.log.gz 2023-12-10T23:40:13.659 INFO:teuthology.orchestra.run.smithi076.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45304.log.gz 2023-12-10T23:40:13.659 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73121.log 2023-12-10T23:40:13.660 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.75766.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.75766.log.gz 2023-12-10T23:40:13.661 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43743.log 2023-12-10T23:40:13.662 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.73121.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.73121.log.gz 2023-12-10T23:40:13.662 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78598.log 2023-12-10T23:40:13.673 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.43743.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42380.log 2023-12-10T23:40:13.674 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.78598.log: 9.2% -- replaced with /var/log/ceph/cluster2-client.mirror.78598.log.gz 2023-12-10T23:40:13.689 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53149.log 2023-12-10T23:40:13.691 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.42380.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.42380.log.gz 2023-12-10T23:40:13.691 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71646.log 2023-12-10T23:40:13.701 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.53149.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68445.log 2023-12-10T23:40:13.702 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.71646.log: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.71646.log.gz 2023-12-10T23:40:13.703 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78716.log 2023-12-10T23:40:13.714 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.68445.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.68445.log.gz 2023-12-10T23:40:13.717 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42256.log 2023-12-10T23:40:13.718 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.78716.log: 9.2% -- replaced with /var/log/ceph/cluster2-client.mirror.78716.log.gz 2023-12-10T23:40:13.737 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.74326.log 2023-12-10T23:40:13.739 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.42256.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52622.log.gz 2023-12-10T23:40:13.740 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47816.log 2023-12-10T23:40:13.743 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.0.74326.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.0.74326.log.gz 2023-12-10T23:40:13.743 INFO:teuthology.orchestra.run.smithi076.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42256.log.gz 2023-12-10T23:40:13.752 INFO:teuthology.orchestra.run.smithi076.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53149.log.gz 2023-12-10T23:40:13.752 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56893.log 2023-12-10T23:40:13.753 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.47816.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.47816.log.gz 2023-12-10T23:40:13.754 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45258.log 2023-12-10T23:40:13.755 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.56893.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56893.log.gz 2023-12-10T23:40:13.755 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54617.log 2023-12-10T23:40:13.756 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.45258.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45258.log.gz 2023-12-10T23:40:13.757 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65340.log 2023-12-10T23:40:13.773 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.54617.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.39911.log 2023-12-10T23:40:13.774 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.65340.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.65340.log.gz 2023-12-10T23:40:13.777 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78853.log 2023-12-10T23:40:13.778 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.2.39911.log: 79.4% -- replaced with /var/log/ceph/cluster1-client.mirror.2.39911.log.gz 2023-12-10T23:40:13.788 INFO:teuthology.orchestra.run.smithi076.stderr: 93.6%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71222.log 2023-12-10T23:40:13.789 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster2-client.mirror.78853.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.78853.log.gz 2023-12-10T23:40:13.789 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.54617.log.gz 2023-12-10T23:40:13.790 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75745.log 2023-12-10T23:40:13.790 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.71222.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71222.log.gz 2023-12-10T23:40:13.791 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cluster1-client.mirror.75745.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.75745.log.gz 2023-12-10T23:40:13.847 INFO:teuthology.orchestra.run.smithi076.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52799.log.gz 2023-12-10T23:40:13.895 INFO:teuthology.orchestra.run.smithi076.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49536.log.gz 2023-12-10T23:40:13.900 INFO:teuthology.orchestra.run.smithi076.stderr: 92.7% -- replaced with /var/log/ceph/cluster1-client.mirror.1.49558.log.gz 2023-12-10T23:40:14.082 INFO:teuthology.orchestra.run.smithi076.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.43743.log.gz 2023-12-10T23:40:17.829 INFO:teuthology.orchestra.run.smithi033.stderr: 94.4% -- replaced with /var/log/ceph/cluster1-osd.1.log.gz 2023-12-10T23:40:19.205 INFO:teuthology.orchestra.run.smithi033.stderr: 94.4% -- replaced with /var/log/ceph/cluster1-osd.2.log.gz 2023-12-10T23:40:19.296 INFO:teuthology.orchestra.run.smithi033.stderr: 94.4% -- replaced with /var/log/ceph/cluster1-osd.0.log.gz 2023-12-10T23:40:19.298 INFO:teuthology.orchestra.run.smithi033.stderr: 2023-12-10T23:40:19.298 INFO:teuthology.orchestra.run.smithi033.stderr:real 0m9.413s 2023-12-10T23:40:19.298 INFO:teuthology.orchestra.run.smithi033.stderr:user 0m27.607s 2023-12-10T23:40:19.298 INFO:teuthology.orchestra.run.smithi033.stderr:sys 0m0.978s 2023-12-10T23:40:55.341 INFO:teuthology.orchestra.run.smithi076.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-osd.1.log.gz 2023-12-10T23:40:59.676 INFO:teuthology.orchestra.run.smithi076.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-osd.2.log.gz 2023-12-10T23:41:05.585 INFO:teuthology.orchestra.run.smithi076.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-osd.0.log.gz 2023-12-10T23:41:05.587 INFO:teuthology.orchestra.run.smithi076.stderr: 2023-12-10T23:41:05.587 INFO:teuthology.orchestra.run.smithi076.stderr:real 0m55.667s 2023-12-10T23:41:05.587 INFO:teuthology.orchestra.run.smithi076.stderr:user 2m43.168s 2023-12-10T23:41:05.587 INFO:teuthology.orchestra.run.smithi076.stderr:sys 0m6.314s 2023-12-10T23:41:05.589 INFO:tasks.ceph:Archiving logs... 2023-12-10T23:41:05.589 DEBUG:teuthology.misc:Transferring archived files from smithi033:/var/log/ceph to /home/teuthworker/archive/dis-2023-12-10_16:10:42-rbd-wip-dis-testing-1-distro-default-smithi/7486362/remote/smithi033/log 2023-12-10T23:41:05.590 DEBUG:teuthology.orchestra.run.smithi033:> sudo tar c -f - -C /var/log/ceph -- . 2023-12-10T23:41:08.597 DEBUG:teuthology.misc:Transferring archived files from smithi076:/var/log/ceph to /home/teuthworker/archive/dis-2023-12-10_16:10:42-rbd-wip-dis-testing-1-distro-default-smithi/7486362/remote/smithi076/log 2023-12-10T23:41:08.598 DEBUG:teuthology.orchestra.run.smithi076:> sudo tar c -f - -C /var/log/ceph -- . 2023-12-10T23:41:28.037 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi033 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json' 2023-12-10T23:41:28.042 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-12-10T23:41:28.057 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing 2023-12-10T23:41:28.064 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-12-10T23:41:28.064 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-10T23:41:28.085 DEBUG:teuthology.orchestra.run.smithi076:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-10T23:41:28.143 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-12-10T23:41:28.154 INFO:teuthology.task.clock:Checking final clock skew... 2023-12-10T23:41:28.155 DEBUG:teuthology.orchestra.run.smithi033:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-10T23:41:28.157 DEBUG:teuthology.orchestra.run.smithi076:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-10T23:41:28.170 INFO:teuthology.orchestra.run.smithi033.stdout: remote refid st t when poll reach delay offset jitter 2023-12-10T23:41:28.170 INFO:teuthology.orchestra.run.smithi033.stdout:============================================================================== 2023-12-10T23:41:28.170 INFO:teuthology.orchestra.run.smithi033.stdout:*hv01.front.sepi 44.4.53.2 2 u 69 128 377 0.143 -1.817 0.061 2023-12-10T23:41:28.170 INFO:teuthology.orchestra.run.smithi033.stdout:+hv02.front.sepi 63.231.80.2 3 u 71 128 377 0.101 +1.196 0.067 2023-12-10T23:41:28.170 INFO:teuthology.orchestra.run.smithi033.stdout:+hv03.front.sepi 74.6.168.72 3 u 59 128 377 0.103 -0.239 0.154 2023-12-10T23:41:28.170 INFO:teuthology.orchestra.run.smithi033.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2023-12-10T23:41:28.176 INFO:teuthology.orchestra.run.smithi076.stdout: remote refid st t when poll reach delay offset jitter 2023-12-10T23:41:28.176 INFO:teuthology.orchestra.run.smithi076.stdout:============================================================================== 2023-12-10T23:41:28.176 INFO:teuthology.orchestra.run.smithi076.stdout:*hv01.front.sepi 44.4.53.2 2 u 223 256 377 0.160 -3.973 1.204 2023-12-10T23:41:28.176 INFO:teuthology.orchestra.run.smithi076.stdout:+hv02.front.sepi 63.231.80.2 3 u 115 256 377 0.182 -0.310 0.597 2023-12-10T23:41:28.176 INFO:teuthology.orchestra.run.smithi076.stdout:+hv03.front.sepi 74.6.168.72 3 u 93 256 377 0.180 -1.777 0.576 2023-12-10T23:41:28.176 INFO:teuthology.orchestra.run.smithi076.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2023-12-10T23:41:28.177 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-12-10T23:41:28.188 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-12-10T23:41:28.188 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-12-10T23:41:28.199 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-12-10T23:41:28.262 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-12-10T23:41:28.273 INFO:teuthology.task.internal:Duration was 18061.281277 seconds 2023-12-10T23:41:28.273 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-12-10T23:41:28.284 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-12-10T23:41:28.284 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-10T23:41:28.287 DEBUG:teuthology.orchestra.run.smithi076:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-10T23:41:28.324 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-12-10T23:41:28.324 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi033.front.sepia.ceph.com 2023-12-10T23:41:28.325 DEBUG:teuthology.orchestra.run.smithi033:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-10T23:41:28.375 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi076.front.sepia.ceph.com 2023-12-10T23:41:28.375 DEBUG:teuthology.orchestra.run.smithi076:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-10T23:41:28.385 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-12-10T23:41:28.385 DEBUG:teuthology.orchestra.run.smithi033:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-10T23:41:28.419 DEBUG:teuthology.orchestra.run.smithi076:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-10T23:41:28.450 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-12-10T23:41:28.450 DEBUG:teuthology.orchestra.run.smithi033:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-10T23:41:28.483 DEBUG:teuthology.orchestra.run.smithi076:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-10T23:41:28.547 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-12-10T23:41:28.559 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-12-10T23:41:28.559 DEBUG:teuthology.orchestra.run.smithi033:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-10T23:41:28.570 DEBUG:teuthology.orchestra.run.smithi076:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-10T23:41:28.580 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-12-10T23:41:28.591 DEBUG:teuthology.orchestra.run.smithi033:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-10T23:41:28.615 DEBUG:teuthology.orchestra.run.smithi076:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-10T23:41:28.625 INFO:teuthology.orchestra.run.smithi033.stdout:kernel.core_pattern = core 2023-12-10T23:41:28.636 INFO:teuthology.orchestra.run.smithi076.stdout:kernel.core_pattern = core 2023-12-10T23:41:28.660 DEBUG:teuthology.orchestra.run.smithi033:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-10T23:41:28.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-10T23:41:28.698 DEBUG:teuthology.orchestra.run.smithi076:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-10T23:41:28.705 WARNING:teuthology.task.internal:Found coredumps on ubuntu@smithi076.front.sepia.ceph.com, flagging run as failed 2023-12-10T23:41:28.706 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-12-10T23:41:28.718 INFO:teuthology.task.internal:Transferring archived files... 2023-12-10T23:41:28.719 DEBUG:teuthology.misc:Transferring archived files from smithi033:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/dis-2023-12-10_16:10:42-rbd-wip-dis-testing-1-distro-default-smithi/7486362/remote/smithi033 2023-12-10T23:41:28.720 DEBUG:teuthology.orchestra.run.smithi033:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-10T23:41:28.764 DEBUG:teuthology.misc:Transferring archived files from smithi076:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/dis-2023-12-10_16:10:42-rbd-wip-dis-testing-1-distro-default-smithi/7486362/remote/smithi076 2023-12-10T23:41:28.764 DEBUG:teuthology.orchestra.run.smithi076:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-10T23:41:59.950 INFO:teuthology.task.internal:Transferring binaries for coredumps... 2023-12-10T23:42:00.409 INFO:teuthology.task.internal: core looks like: /home/teuthworker/archive/dis-2023-12-10_16:10:42-rbd-wip-dis-testing-1-distro-default-smithi/7486362/remote/smithi076/coredump/1702237137.62748.core.gz: gzip compressed data, was "1702237137.62748.core", last modified: Sun Dec 10 23:41:44 2023, max compression, original size modulo 2^32 645836800 2023-12-10T23:42:00.410 INFO:teuthology.task.internal:core is compressed, try accessing gzip file ... 2023-12-10T23:42:02.770 INFO:teuthology.task.internal: core looks like: /tmp/tmpidod6dh7: ELF 64-bit LSB core file, x86-64, version 1 (SYSV), SVR4-style, from 'rbd-mirror --foreground --cluster cluster2 --id mirror.1', real uid: 1000, effective uid: 1000, real gid: 1271, effective gid: 1271, execfn: '/usr/bin/rbd-mirror', platform: 'x86_64' 2023-12-10T23:42:02.831 INFO:teuthology.task.internal: dump_program: rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-12-10T23:42:02.831 DEBUG:teuthology.orchestra.run.smithi076:> which 'rbd-mirror --foreground --cluster cluster2 --id mirror.1' 2023-12-10T23:42:02.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-10T23:42:02.839 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/internal/__init__.py", line 392, in archive yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/internal/__init__.py", line 420, in archive fetch_binaries_for_coredumps(path, rem) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/internal/__init__.py", line 345, in fetch_binaries_for_coredumps remote_path = remote.sh(['which', dump_program]).rstrip() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi076 with status 1: "which 'rbd-mirror --foreground --cluster cluster2 --id mirror.1'" 2023-12-10T23:42:02.851 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-12-10T23:42:02.867 INFO:teuthology.task.internal:Not uploading archives. 2023-12-10T23:42:02.867 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-12-10T23:42:02.879 INFO:teuthology.task.internal:Tidying up after the test... 2023-12-10T23:42:02.879 DEBUG:teuthology.orchestra.run.smithi033:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-10T23:42:02.885 DEBUG:teuthology.orchestra.run.smithi076:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-10T23:42:02.890 INFO:teuthology.orchestra.run.smithi033.stdout: 655472 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Dec 10 23:41 /home/ubuntu/cephtest 2023-12-10T23:42:02.890 INFO:teuthology.orchestra.run.smithi033.stdout: 655473 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Dec 10 23:41 /home/ubuntu/cephtest/archive 2023-12-10T23:42:02.890 INFO:teuthology.orchestra.run.smithi033.stdout: 658631 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 10 23:41 /home/ubuntu/cephtest/archive/syslog 2023-12-10T23:42:02.890 INFO:teuthology.orchestra.run.smithi033.stdout: 658633 4 -rw-rw-rw- 1 ubuntu ubuntu 774 Dec 10 23:40 /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2023-12-10T23:42:02.890 INFO:teuthology.orchestra.run.smithi033.stdout: 658632 96 -rw-rw-r-- 1 ubuntu ubuntu 96669 Dec 10 23:41 /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-10T23:42:02.890 INFO:teuthology.orchestra.run.smithi033.stdout: 655433 80 -rw-rw-rw- 1 ubuntu ubuntu 81558 Dec 10 23:41 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2023-12-10T23:42:02.891 INFO:teuthology.orchestra.run.smithi076.stdout: 655472 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Dec 10 23:41 /home/ubuntu/cephtest 2023-12-10T23:42:02.891 INFO:teuthology.orchestra.run.smithi076.stdout: 674216 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Dec 10 18:51 /home/ubuntu/cephtest/mnt.cluster1.mirror 2023-12-10T23:42:02.891 INFO:teuthology.orchestra.run.smithi076.stdout: 674217 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Dec 10 18:52 /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2023-12-10T23:42:02.891 INFO:teuthology.orchestra.run.smithi076.stdout: 688372 4 drwxrwxr-x 2 ubuntu ubuntu 4096 Dec 10 18:52 /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp 2023-12-10T23:42:02.891 INFO:teuthology.orchestra.run.smithi076.stdout: 655473 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Dec 10 18:40 /home/ubuntu/cephtest/archive 2023-12-10T23:42:02.892 INFO:teuthology.orchestra.run.smithi076.stdout: 658625 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 10 23:41 /home/ubuntu/cephtest/archive/syslog 2023-12-10T23:42:02.892 INFO:teuthology.orchestra.run.smithi076.stdout: 658627 4 -rw-rw-rw- 1 ubuntu ubuntu 777 Dec 10 23:38 /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2023-12-10T23:42:02.892 INFO:teuthology.orchestra.run.smithi076.stdout: 658626 96 -rw-rw-r-- 1 ubuntu ubuntu 97654 Dec 10 23:41 /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-10T23:42:02.892 INFO:teuthology.orchestra.run.smithi076.stdout: 663690 84 -rw-rw-rw- 1 ubuntu ubuntu 83751 Dec 10 23:41 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2023-12-10T23:42:02.892 INFO:teuthology.orchestra.run.smithi076.stdout: 658624 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 10 19:41 /home/ubuntu/cephtest/archive/coredump 2023-12-10T23:42:02.892 INFO:teuthology.orchestra.run.smithi076.stdout: 690004 46364 -rw------- 1 ubuntu ubuntu 634953728 Dec 10 19:41 /home/ubuntu/cephtest/archive/coredump/1702237287.72691.core 2023-12-10T23:42:02.892 INFO:teuthology.orchestra.run.smithi076.stdout: 690002 50692 -rw------- 1 ubuntu ubuntu 645836800 Dec 10 19:38 /home/ubuntu/cephtest/archive/coredump/1702237137.62748.core 2023-12-10T23:42:02.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-10T23:42:02.893 INFO:teuthology.orchestra.run.smithi033.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-12-10T23:42:02.893 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_3d2db44fa991bc355074aa9b7647f63b77201fef/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi033 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-12-10T23:42:02.900 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-12-10T23:42:02.943 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-12-10T23:42:02.956 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: RuntimeError: error during thrashing 2023-12-10T23:42:02.956 INFO:teuthology.nuke:Checking targets against current locks 2023-12-10T23:42:02.962 INFO:teuthology.orchestra.run.smithi076.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-12-10T23:42:02.986 DEBUG:teuthology.nuke:shortname: smithi033 2023-12-10T23:42:02.987 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-10T23:42:03.010 DEBUG:teuthology.nuke:shortname: smithi076 2023-12-10T23:42:03.011 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-10T23:42:03.013 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi033.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2023-12-10_16:10:42-rbd-wip-dis-testing-1-distro-default-smithi/7486362', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-10 18:30:59.830640', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCg4fpIf9BgAE0K5vY3eaCYNnAdpFkPYg83UNzpXJx+VdnMglbkcDNn9oNHV0h8GlQ68zOzCcQCAjDqUW/H3wNI='} 2023-12-10T23:42:03.034 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi076.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2023-12-10_16:10:42-rbd-wip-dis-testing-1-distro-default-smithi/7486362', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-10 18:30:59.831797', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDz6mTNSmU7E4IwEUFsmyAfIf0TTvzaqidmq2OKROs5pSoYL6ng2SXq1yuh1NB7AMqvNCBdduJ5kk6F5IVYNexs='} 2023-12-10T23:42:03.079 INFO:teuthology.orchestra.console.smithi033:Power off 2023-12-10T23:42:03.079 DEBUG:teuthology.orchestra.console.smithi033:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-10T23:42:03.117 INFO:teuthology.orchestra.console.smithi076:Power off 2023-12-10T23:42:03.117 DEBUG:teuthology.orchestra.console.smithi076:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-10T23:42:03.139 DEBUG:teuthology.orchestra.console.smithi076:power off output: Chassis Power Control: Down/Off 2023-12-10T23:42:03.140 DEBUG:teuthology.orchestra.console.smithi076:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-10T23:42:03.146 DEBUG:teuthology.orchestra.console.smithi033:power off output: Chassis Power Control: Down/Off 2023-12-10T23:42:03.146 DEBUG:teuthology.orchestra.console.smithi033:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-10T23:42:03.160 DEBUG:teuthology.orchestra.console.smithi076:check power output: Chassis Power is on 2023-12-10T23:42:03.169 DEBUG:teuthology.orchestra.console.smithi033:check power output: Chassis Power is on 2023-12-10T23:42:07.161 DEBUG:teuthology.orchestra.console.smithi076:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-10T23:42:07.169 DEBUG:teuthology.orchestra.console.smithi033:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-10T23:42:07.271 DEBUG:teuthology.orchestra.console.smithi076:check power output: Chassis Power is on 2023-12-10T23:42:07.277 DEBUG:teuthology.orchestra.console.smithi033:check power output: Chassis Power is on 2023-12-10T23:42:11.276 DEBUG:teuthology.orchestra.console.smithi076:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-10T23:42:11.286 DEBUG:teuthology.orchestra.console.smithi033:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-10T23:42:11.388 DEBUG:teuthology.orchestra.console.smithi076:check power output: Chassis Power is on 2023-12-10T23:42:11.396 DEBUG:teuthology.orchestra.console.smithi033:check power output: Chassis Power is off 2023-12-10T23:42:11.497 INFO:teuthology.orchestra.console.smithi033:Power off completed 2023-12-10T23:42:15.393 DEBUG:teuthology.orchestra.console.smithi076:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-10T23:42:15.505 DEBUG:teuthology.orchestra.console.smithi076:check power output: Chassis Power is off 2023-12-10T23:42:15.606 INFO:teuthology.orchestra.console.smithi076:Power off completed 2023-12-10T23:42:15.713 INFO:teuthology.run:Summary data: description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-stupid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-stress-workunit-minimum} duration: 18061.281277418137 failure_reason: 'Command failed (workunit test rbd/rbd_mirror_stress.sh) on smithi076 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3d2db44fa991bc355074aa9b7647f63b77201fef TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\''\'' MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering RBD_MIRROR_INSTANCES=4 RBD_MIRROR_MODE=snapshot RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh''' flavor: default owner: scheduled_dis@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=e6bc631a89974d6b8856b0f0c5b2243f status: fail success: false 2023-12-10T23:42:15.713 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-10T23:42:15.798 INFO:teuthology.run:FAIL